| 2025-11-22 18:00:28,752 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-22 18:00:28,762 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json |
| 2025-11-22 18:00:28,763 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 18:00:28,763 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 18:00:28,766 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-22 18:00:28,766 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-22 18:00:28,767 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 18:00:28,767 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-22 18:00:28,767 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-22 18:00:28,768 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-22 18:00:28,768 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-22 18:00:28,769 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-22 18:00:28,783 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-22 18:00:28,785 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:00:29,963 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-22 18:00:30,946 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_180030 |
| 2025-11-22 18:00:30,947 - __main__ - INFO - Run ID: 5daca621f4734a9d83a4026d35c5be11 |
| 2025-11-22 18:00:30,948 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:00:31,382 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-22 18:00:31,384 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-22 18:00:31,394 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-22 18:00:31,395 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-22 18:00:31,396 - __main__ - INFO - |
| 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 18:00:39,649 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-22 18:00:39,649 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-22 18:00:39,688 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file |
| 2025-11-22 18:00:40,202 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file |
| 2025-11-22 18:00:40,215 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl |
| 2025-11-22 18:00:40,813 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) |
| 2025-11-22 18:00:44,710 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 |
| 2025-11-22 18:00:47,919 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 |
| 2025-11-22 18:00:48,496 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) |
| 2025-11-22 18:00:48,503 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl |
| 2025-11-22 18:00:53,190 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) |
| 2025-11-22 18:00:57,043 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:00,317 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:03,440 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:06,523 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:09,601 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:12,744 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:15,855 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:18,947 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:21,980 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:25,052 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:28,125 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:31,191 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:34,271 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:37,341 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:40,328 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:43,407 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:46,462 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:49,521 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:52,563 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:55,616 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 |
| 2025-11-22 18:01:58,692 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:01,773 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:04,835 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:07,850 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:10,864 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:13,929 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:16,981 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:20,014 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:23,050 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:02:25,312 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) |
| 2025-11-22 18:02:25,370 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 |
| 2025-11-22 18:02:25,371 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-22 18:02:25,372 - __main__ - INFO - Sample 1: |
| 2025-11-22 18:02:25,372 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... |
| 2025-11-22 18:02:25,373 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... |
| 2025-11-22 18:02:25,373 - __main__ - INFO - Sample 2: |
| 2025-11-22 18:02:25,374 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... |
| 2025-11-22 18:02:25,375 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... |
| 2025-11-22 18:02:25,376 - __main__ - INFO - Sample 3: |
| 2025-11-22 18:02:25,378 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... |
| 2025-11-22 18:02:25,378 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience |
| 2025-11-22 18:02:29,507 - __main__ - INFO - Filtering statistics: {'kept': 3190457} |
| 2025-11-22 18:02:29,519 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 |
| 2025-11-22 18:02:32,819 - __main__ - INFO - Removed 116 duplicates |
| 2025-11-22 18:02:32,990 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 |
| 2025-11-22 18:02:38,144 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 18:02:38,144 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-22 18:02:38,145 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 18:02:38,146 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 18:02:38,146 - __main__ - INFO - Test: 159,518 samples |
| 2025-11-22 18:02:38,147 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 18:02:38,518 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-22 18:05:00,120 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-22 18:05:00,123 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 18:05:00,123 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 18:05:00,124 - __main__ - INFO - Test: 159,518 samples |
|
|
| 2025-11-22 18:05:02,062 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-22 18:05:02,275 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-22 18:05:03,804 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-22 18:05:10,989 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-22 18:05:12,932 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-22 18:05:12,936 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) |
| 2025-11-22 18:05:12,937 - __main__ - INFO - Next run will skip tokenization and load from cache |
| 2025-11-22 18:05:12,938 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... |
| 2025-11-22 18:06:05,795 - __main__ - ERROR - |
| ❌ Pipeline failed: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. |
| Traceback (most recent call last): |
| File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1987, in main |
| trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1330, in __init__ |
| self.tokenized_datasets = self.dataset_dict.map( |
| ^^^^^^^^^^^^^^^^^^^^^^ |
| File "/root/venv/lib/python3.12/site-packages/datasets/dataset_dict.py", line 953, in map |
| dataset_dict[split] = dataset.map( |
| ^^^^^^^^^^^^ |
| File "/root/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 562, in wrapper |
| out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/root/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3332, in map |
| for rank, done, content in iflatmap_unordered( |
| File "/root/venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 619, in iflatmap_unordered |
| raise RuntimeError( |
| RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. |
| 2025-11-22 18:06:07,362 - __main__ - INFO - ✅ MLflow run ended: 5daca621f4734a9d83a4026d35c5be11 |
| 2025-11-22 18:09:39,437 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-22 18:09:39,447 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json |
| 2025-11-22 18:09:39,447 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 18:09:39,448 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 18:09:39,450 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-22 18:09:39,451 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-22 18:09:39,451 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 18:09:39,452 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-22 18:09:39,452 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-22 18:09:39,453 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-22 18:09:39,453 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-22 18:09:39,453 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-22 18:09:39,471 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-22 18:09:39,472 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:09:40,614 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-22 18:09:41,671 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_180941 |
| 2025-11-22 18:09:41,673 - __main__ - INFO - Run ID: e2133803031f4db38199a0e4895e1a60 |
| 2025-11-22 18:09:41,674 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:09:42,139 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-22 18:09:42,145 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-22 18:09:42,155 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-22 18:09:42,156 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-22 18:09:42,156 - __main__ - INFO - |
| 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 18:09:45,323 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-22 18:09:45,324 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-22 18:09:45,411 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file |
| 2025-11-22 18:09:45,926 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file |
| 2025-11-22 18:09:45,927 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl |
| 2025-11-22 18:09:46,151 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) |
| 2025-11-22 18:09:49,794 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 |
| 2025-11-22 18:09:52,995 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 |
| 2025-11-22 18:09:53,438 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) |
| 2025-11-22 18:09:53,442 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl |
| 2025-11-22 18:09:54,507 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) |
| 2025-11-22 18:09:57,526 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:00,517 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:03,537 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:06,547 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:09,537 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:12,525 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:15,566 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:18,576 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:21,624 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:24,650 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:27,674 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:30,691 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:33,741 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:36,783 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:39,789 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:42,854 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:10:45,927 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:11:30,196 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-22 18:11:30,219 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json |
| 2025-11-22 18:11:30,220 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 18:11:30,221 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 18:11:30,226 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-22 18:11:30,226 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-22 18:11:30,227 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 18:11:30,227 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-22 18:11:30,228 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-22 18:11:30,228 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-22 18:11:30,228 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-22 18:11:30,229 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-22 18:11:30,248 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-22 18:11:30,248 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:11:31,402 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-22 18:11:32,515 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_181131 |
| 2025-11-22 18:11:32,516 - __main__ - INFO - Run ID: 1ea1fc3d7e6841139ec6d2af5ac6f07b |
| 2025-11-22 18:11:32,517 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:11:32,984 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-22 18:11:32,986 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-22 18:11:32,992 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-22 18:11:32,992 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-22 18:11:32,993 - __main__ - INFO - |
| 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 18:11:35,172 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-22 18:11:35,173 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-22 18:11:35,211 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file |
| 2025-11-22 18:11:35,717 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file |
| 2025-11-22 18:11:35,718 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl |
| 2025-11-22 18:11:35,859 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) |
| 2025-11-22 18:11:39,616 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 |
| 2025-11-22 18:11:42,804 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 |
| 2025-11-22 18:11:43,278 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) |
| 2025-11-22 18:11:43,285 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl |
| 2025-11-22 18:11:43,482 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) |
| 2025-11-22 18:11:46,608 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:11:49,715 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:11:52,824 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:11:55,973 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:11:59,154 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:02,267 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:05,413 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:08,545 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:11,668 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:14,810 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:17,960 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:21,093 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:24,516 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:27,835 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:30,928 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:34,075 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:37,224 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:40,407 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:43,527 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:46,651 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:49,754 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:52,862 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:55,955 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:12:59,034 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:13:02,074 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:13:05,178 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:13:08,270 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:13:11,352 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:13:14,433 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:13:16,768 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) |
| 2025-11-22 18:13:16,819 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 |
| 2025-11-22 18:13:16,819 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-22 18:13:16,820 - __main__ - INFO - Sample 1: |
| 2025-11-22 18:13:16,821 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... |
| 2025-11-22 18:13:16,821 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... |
| 2025-11-22 18:13:16,822 - __main__ - INFO - Sample 2: |
| 2025-11-22 18:13:16,822 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... |
| 2025-11-22 18:13:16,822 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... |
| 2025-11-22 18:13:16,823 - __main__ - INFO - Sample 3: |
| 2025-11-22 18:13:16,823 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... |
| 2025-11-22 18:13:16,824 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience |
| 2025-11-22 18:13:20,795 - __main__ - INFO - Filtering statistics: {'kept': 3190457} |
| 2025-11-22 18:13:20,796 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 |
| 2025-11-22 18:13:24,142 - __main__ - INFO - Removed 116 duplicates |
| 2025-11-22 18:13:24,311 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 |
| 2025-11-22 18:13:29,401 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 18:13:29,418 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-22 18:13:29,419 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 18:13:29,419 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 18:13:29,420 - __main__ - INFO - Test: 159,518 samples |
| 2025-11-22 18:13:29,421 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 18:13:29,781 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-22 18:15:50,493 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-22 18:15:50,495 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 18:15:50,496 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 18:15:50,496 - __main__ - INFO - Test: 159,518 samples |
|
|
| 2025-11-22 18:15:52,526 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) |
| 2025-11-22 18:15:52,527 - __main__ - INFO - Next run will skip tokenization and load from cache |
| 2025-11-22 18:15:52,528 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... |
| 2025-11-22 18:16:28,894 - __main__ - ERROR - |
| ❌ Pipeline failed: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. |
| Traceback (most recent call last): |
| File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1990, in main |
| trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1329, in __init__ |
| self.tokenized_datasets = self.dataset_dict.map( |
| ^^^^^^^^^^^^^^^^^^^^^^ |
| File "/root/venv/lib/python3.12/site-packages/datasets/dataset_dict.py", line 953, in map |
| dataset_dict[split] = dataset.map( |
| ^^^^^^^^^^^^ |
| File "/root/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 562, in wrapper |
| out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/root/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3332, in map |
| for rank, done, content in iflatmap_unordered( |
| File "/root/venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 619, in iflatmap_unordered |
| raise RuntimeError( |
| RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. |
| 2025-11-22 18:16:30,480 - __main__ - INFO - ✅ MLflow run ended: 1ea1fc3d7e6841139ec6d2af5ac6f07b |
| 2025-11-22 18:32:47,906 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-22 18:32:47,938 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json |
| 2025-11-22 18:32:47,939 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 18:32:47,939 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 18:32:47,942 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-22 18:32:47,943 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-22 18:32:47,943 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 18:32:47,944 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-22 18:32:47,944 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-22 18:32:47,946 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-22 18:32:47,946 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-22 18:32:47,947 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-22 18:32:47,963 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-22 18:32:47,964 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:32:48,787 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-22 18:32:50,249 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_183249 |
| 2025-11-22 18:32:50,251 - __main__ - INFO - Run ID: 4bd900a24a6e48ac94ba57dbfabfade2 |
| 2025-11-22 18:32:50,251 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:32:50,617 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-22 18:32:50,621 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-22 18:32:50,637 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-22 18:32:50,638 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-22 18:32:50,639 - __main__ - INFO - |
| 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 18:32:52,949 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-22 18:32:52,950 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-22 18:32:52,990 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file |
| 2025-11-22 18:32:53,498 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file |
| 2025-11-22 18:32:53,499 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl |
| 2025-11-22 18:32:53,780 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) |
| 2025-11-22 18:32:57,511 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 |
| 2025-11-22 18:33:00,602 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 |
| 2025-11-22 18:33:01,086 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) |
| 2025-11-22 18:33:01,090 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl |
| 2025-11-22 18:33:02,496 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) |
| 2025-11-22 18:33:05,650 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:08,753 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:11,895 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:15,188 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:18,337 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:21,451 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:24,913 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:28,019 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:31,323 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:34,635 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:37,935 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:41,203 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:44,494 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:47,922 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:51,815 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:55,183 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:33:58,359 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:01,528 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:04,728 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:07,921 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:11,102 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:14,245 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:17,409 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:20,593 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:23,682 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:26,823 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:29,975 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:33,136 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:36,287 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 |
| 2025-11-22 18:34:38,668 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) |
| 2025-11-22 18:34:38,717 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 |
| 2025-11-22 18:34:38,718 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-22 18:34:38,719 - __main__ - INFO - Sample 1: |
| 2025-11-22 18:34:38,720 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... |
| 2025-11-22 18:34:38,721 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... |
| 2025-11-22 18:34:38,721 - __main__ - INFO - Sample 2: |
| 2025-11-22 18:34:38,722 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... |
| 2025-11-22 18:34:38,722 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... |
| 2025-11-22 18:34:38,723 - __main__ - INFO - Sample 3: |
| 2025-11-22 18:34:38,723 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... |
| 2025-11-22 18:34:38,724 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience |
| 2025-11-22 18:34:42,893 - __main__ - INFO - Filtering statistics: {'kept': 3190457} |
| 2025-11-22 18:34:42,894 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 |
| 2025-11-22 18:34:46,294 - __main__ - INFO - Removed 116 duplicates |
| 2025-11-22 18:34:46,468 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 |
| 2025-11-22 18:34:51,669 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 18:34:51,670 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-22 18:34:51,671 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 18:34:51,671 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 18:34:51,672 - __main__ - INFO - Test: 159,518 samples |
| 2025-11-22 18:34:51,672 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 18:34:52,022 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-22 18:37:19,923 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-22 18:37:19,926 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 18:37:19,927 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 18:37:19,927 - __main__ - INFO - Test: 159,518 samples |
|
|
| 2025-11-22 18:37:21,281 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) |
| 2025-11-22 18:37:21,281 - __main__ - INFO - Next run will skip tokenization and load from cache |
| 2025-11-22 18:37:21,282 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... |
| 2025-11-22 18:37:53,621 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-22 18:37:53,623 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-22 18:37:53,732 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-22 18:37:54,829 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-22 18:38:01,229 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-22 18:38:01,704 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-22 18:38:01,723 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-22 18:38:01,863 - __main__ - INFO - ✅ Trainer initialized |
| 2025-11-22 18:38:01,864 - __main__ - INFO - |
| 🔬 STEP 1: Baseline Evaluation |
| 2025-11-22 18:38:02,507 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 |
| 2025-11-22 18:38:02,515 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 18:38:02,517 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... |
| 2025-11-22 18:38:02,517 - __main__ - INFO - ====================================================================== |
| 2025-11-22 18:38:23,653 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.0198 |
| 2025-11-22 18:38:23,662 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-22 18:38:23,715 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... |
| 2025-11-22 18:38:23,717 - __main__ - INFO - Batch 1/16... |
| 2025-11-22 18:38:25,543 - __main__ - INFO - Batch 5/16... |
| 2025-11-22 18:38:27,277 - __main__ - INFO - Batch 10/16... |
| 2025-11-22 18:38:28,933 - __main__ - INFO - Batch 15/16... |
| 2025-11-22 18:38:29,534 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-22 18:38:29,545 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-22 18:38:29,554 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-22 18:38:29,555 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-22 18:38:29,557 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision |
| 2025-11-22 18:38:29,558 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-22 18:38:32,978 - __main__ - INFO - ✅ COMET-QE: 0.0383 ± 0.1441 |
| 2025-11-22 18:38:32,979 - __main__ - INFO - |
| ✅ Baseline Results (Test Set): |
| 2025-11-22 18:38:32,980 - __main__ - INFO - BLEU: 0.0198 |
| 2025-11-22 18:38:32,981 - __main__ - INFO - chrF: 17.59 |
| 2025-11-22 18:38:32,981 - __main__ - INFO - COMET: 0.0383 |
| 2025-11-22 18:38:32,982 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 18:38:32,982 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 18:38:32,983 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-22 18:38:32,983 - __main__ - INFO - ====================================================================== |
| 2025-11-22 18:38:32,983 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-22 18:38:32,987 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-22 18:38:34,322 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-22 18:38:34,394 - __main__ - INFO - ✅ Domain chrF: 0.93 |
| 2025-11-22 18:38:34,395 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-22 18:38:34,420 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-22 18:38:34,429 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-22 18:38:34,435 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-22 18:38:34,438 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-22 18:38:36,561 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 |
| 2025-11-22 18:38:36,562 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-22 18:38:36,562 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-22 18:38:36,563 - __main__ - INFO - chrF: 0.93 |
| 2025-11-22 18:38:36,563 - __main__ - INFO - COMET-QE: 0.0000 |
| 2025-11-22 18:38:36,564 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 18:38:36,953 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 |
| 2025-11-22 18:38:36,969 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase1/baseline_results.json |
| 2025-11-22 18:38:36,976 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics |
| 2025-11-22 18:38:36,976 - __main__ - INFO - |
| 🚀 STEP 2: Fine-tuning |
| 2025-11-22 18:38:36,977 - __main__ - INFO - Metrics are being logged to MLflow in real-time |
| 2025-11-22 18:38:36,978 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 18:38:36,979 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1/logs |
| 2025-11-22 18:38:36,979 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') |
| 2025-11-22 18:38:36,981 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 18:38:36,981 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 18:38:36,982 - __main__ - INFO - 🚀 Starting training... |
| 2025-11-22 18:38:36,983 - __main__ - INFO - ====================================================================== |
| 2025-11-22 18:38:38,036 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) |
| 2025-11-22 18:39:02,025 - __main__ - INFO - 📈 Epoch progress: 0.01/8 (0.1%) |
| 2025-11-22 18:39:25,763 - __main__ - INFO - 📈 Epoch progress: 0.02/8 (0.2%) |
| 2025-11-22 18:39:50,193 - __main__ - INFO - 📈 Epoch progress: 0.03/8 (0.4%) |
| 2025-11-22 18:40:14,121 - __main__ - INFO - 📈 Epoch progress: 0.04/8 (0.5%) |
| 2025-11-22 18:40:38,244 - __main__ - INFO - 📈 Epoch progress: 0.05/8 (0.6%) |
| 2025-11-22 18:41:10,169 - __main__ - INFO - 📈 Epoch progress: 0.06/8 (0.7%) |
| 2025-11-22 18:41:40,768 - __main__ - INFO - 📈 Epoch progress: 0.07/8 (0.8%) |
| 2025-11-22 18:42:11,121 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (0.9%) |
| 2025-11-22 18:42:41,739 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (1.1%) |
| 2025-11-22 18:43:12,016 - __main__ - INFO - 📈 Epoch progress: 0.09/8 (1.2%) |
| 2025-11-22 19:10:25,471 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-22 19:10:25,481 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json |
| 2025-11-22 19:10:25,481 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 19:10:25,482 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 19:10:25,484 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-22 19:10:25,484 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-22 19:10:25,485 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 19:10:25,485 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-22 19:10:25,486 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-22 19:10:25,486 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-22 19:10:25,487 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-22 19:10:25,487 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-22 19:10:25,503 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-22 19:10:25,517 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:10:26,702 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-22 19:10:27,695 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_191027 |
| 2025-11-22 19:10:27,696 - __main__ - INFO - Run ID: bdc7acccde764a5f94d2bfab879011bc |
| 2025-11-22 19:10:27,697 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:10:28,059 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-22 19:10:28,062 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-22 19:10:28,072 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-22 19:10:28,073 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-22 19:10:28,073 - __main__ - INFO - |
| 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 19:10:30,734 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-22 19:10:30,735 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-22 19:10:30,774 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file |
| 2025-11-22 19:10:31,285 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file |
| 2025-11-22 19:10:31,289 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl |
| 2025-11-22 19:10:31,550 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) |
| 2025-11-22 19:10:35,181 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 |
| 2025-11-22 19:11:10,118 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-22 19:11:10,126 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json |
| 2025-11-22 19:11:10,126 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 19:11:10,127 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 19:11:10,129 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-22 19:11:10,129 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-22 19:11:10,130 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 19:11:10,130 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-22 19:11:10,131 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-22 19:11:10,131 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-22 19:11:10,132 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-22 19:11:10,132 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-22 19:11:10,152 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-22 19:11:10,153 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:11:10,578 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-22 19:11:11,452 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_191110 |
| 2025-11-22 19:11:11,454 - __main__ - INFO - Run ID: b71909a6bb4441a98279b91c6d4fd466 |
| 2025-11-22 19:11:11,454 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:11:11,821 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow |
| 2025-11-22 19:11:11,823 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-22 19:11:11,830 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-22 19:11:11,831 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-22 19:11:11,831 - __main__ - INFO - |
| 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 19:11:13,547 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-22 19:11:13,548 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-22 19:11:13,588 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file |
| 2025-11-22 19:11:14,104 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file |
| 2025-11-22 19:11:14,114 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl |
| 2025-11-22 19:11:14,258 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) |
| 2025-11-22 19:11:18,111 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 |
| 2025-11-22 19:11:21,459 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 |
| 2025-11-22 19:11:21,919 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) |
| 2025-11-22 19:11:21,924 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl |
| 2025-11-22 19:11:23,966 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) |
| 2025-11-22 19:11:27,726 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:30,808 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:33,880 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:36,898 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:39,913 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:42,928 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:45,982 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:48,978 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:52,010 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:55,087 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 |
| 2025-11-22 19:11:58,183 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:01,268 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:04,380 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:07,505 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:10,549 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:13,640 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:16,770 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:19,953 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:23,052 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:26,150 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:29,224 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:32,306 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:35,411 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:38,474 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:41,509 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:44,604 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:47,696 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:50,792 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:53,890 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:12:56,211 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) |
| 2025-11-22 19:12:56,262 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 |
| 2025-11-22 19:12:56,263 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-22 19:12:56,264 - __main__ - INFO - Sample 1: |
| 2025-11-22 19:12:56,264 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... |
| 2025-11-22 19:12:56,265 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... |
| 2025-11-22 19:12:56,265 - __main__ - INFO - Sample 2: |
| 2025-11-22 19:12:56,266 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... |
| 2025-11-22 19:12:56,266 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... |
| 2025-11-22 19:12:56,267 - __main__ - INFO - Sample 3: |
| 2025-11-22 19:12:56,267 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... |
| 2025-11-22 19:12:56,267 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience |
| 2025-11-22 19:13:00,365 - __main__ - INFO - Filtering statistics: {'kept': 3190457} |
| 2025-11-22 19:13:00,366 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 |
| 2025-11-22 19:13:03,687 - __main__ - INFO - Removed 116 duplicates |
| 2025-11-22 19:13:03,864 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 |
| 2025-11-22 19:13:09,098 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 19:13:09,115 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-22 19:13:09,117 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 19:13:09,118 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 19:13:09,118 - __main__ - INFO - Test: 159,518 samples |
| 2025-11-22 19:13:09,119 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 19:13:09,479 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-22 19:15:30,511 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-22 19:15:30,520 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 19:15:30,521 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 19:15:30,522 - __main__ - INFO - Test: 159,518 samples |
|
|
| 2025-11-22 19:15:31,951 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) |
| 2025-11-22 19:15:31,952 - __main__ - INFO - Next run will skip tokenization and load from cache |
| 2025-11-22 19:15:31,953 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... |
| 2025-11-22 19:16:05,189 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-22 19:16:05,191 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-22 19:16:05,318 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-22 19:16:06,472 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-22 19:16:13,255 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-22 19:16:13,713 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-22 19:16:13,714 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-22 19:16:13,716 - __main__ - ERROR - |
| ❌ Pipeline failed: --load_best_model_at_end requires the saving steps to be a round multiple of the evaluation steps, but found 1000, which is not a round multiple of 2000. |
| Traceback (most recent call last): |
| File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1990, in main |
| trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1345, in __init__ |
| self._setup_trainer() |
| File "/workspace/franklin/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1371, in _setup_trainer |
| training_args = Seq2SeqTrainingArguments( |
| ^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "<string>", line 140, in __init__ |
| File "/root/venv/lib/python3.12/site-packages/transformers/training_args.py", line 1709, in __post_init__ |
| raise ValueError( |
| ValueError: --load_best_model_at_end requires the saving steps to be a round multiple of the evaluation steps, but found 1000, which is not a round multiple of 2000. |
| 2025-11-22 19:16:15,287 - __main__ - INFO - ✅ MLflow run ended: b71909a6bb4441a98279b91c6d4fd466 |
| 2025-11-22 19:21:07,378 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-22 19:21:07,387 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json |
| 2025-11-22 19:21:07,388 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 19:21:07,388 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 19:21:07,390 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-22 19:21:07,391 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-22 19:21:07,391 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 19:21:07,392 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-22 19:21:07,392 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-22 19:21:07,393 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-22 19:21:07,393 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-22 19:21:07,394 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-22 19:21:07,411 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-22 19:21:07,414 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:21:08,151 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-22 19:21:09,160 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_192108 |
| 2025-11-22 19:21:09,162 - __main__ - INFO - Run ID: 1832aa44606e4efbaecf3870d9bab685 |
| 2025-11-22 19:21:09,162 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:21:09,658 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow |
| 2025-11-22 19:21:09,779 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-22 19:21:09,798 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-22 19:21:09,798 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-22 19:21:09,814 - __main__ - INFO - |
| 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 19:21:11,918 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-22 19:21:11,919 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-22 19:21:11,957 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file |
| 2025-11-22 19:21:12,465 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file |
| 2025-11-22 19:21:12,466 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl |
| 2025-11-22 19:21:12,699 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) |
| 2025-11-22 19:21:16,476 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 |
| 2025-11-22 19:21:20,071 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 |
| 2025-11-22 19:21:20,529 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) |
| 2025-11-22 19:21:20,532 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl |
| 2025-11-22 19:21:22,650 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) |
| 2025-11-22 19:21:25,817 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:28,834 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:31,885 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:34,948 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:37,976 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:40,997 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:44,060 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:47,079 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:50,055 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:53,049 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:56,050 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:21:59,055 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:02,074 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:05,077 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:08,050 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:11,082 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:14,106 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:17,112 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:20,199 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:23,273 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:26,310 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:29,364 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:32,401 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:35,454 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:38,432 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:41,458 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:44,469 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:47,487 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:50,507 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:22:52,804 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) |
| 2025-11-22 19:22:52,874 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 |
| 2025-11-22 19:22:52,875 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-22 19:22:52,875 - __main__ - INFO - Sample 1: |
| 2025-11-22 19:22:52,876 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... |
| 2025-11-22 19:22:52,876 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... |
| 2025-11-22 19:22:52,877 - __main__ - INFO - Sample 2: |
| 2025-11-22 19:22:52,877 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... |
| 2025-11-22 19:22:52,878 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... |
| 2025-11-22 19:22:52,879 - __main__ - INFO - Sample 3: |
| 2025-11-22 19:22:52,879 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... |
| 2025-11-22 19:22:52,880 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience |
| 2025-11-22 19:22:57,109 - __main__ - INFO - Filtering statistics: {'kept': 3190457} |
| 2025-11-22 19:22:57,114 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 |
| 2025-11-22 19:23:00,518 - __main__ - INFO - Removed 116 duplicates |
| 2025-11-22 19:23:00,691 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 |
| 2025-11-22 19:23:05,980 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 19:23:05,981 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-22 19:23:05,982 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 19:23:05,982 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 19:23:05,983 - __main__ - INFO - Test: 159,518 samples |
| 2025-11-22 19:23:05,983 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 19:23:06,360 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-22 19:25:35,062 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-22 19:25:35,065 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 19:25:35,066 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 19:25:35,067 - __main__ - INFO - Test: 159,518 samples |
|
|
| 2025-11-22 19:25:36,537 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) |
| 2025-11-22 19:25:36,538 - __main__ - INFO - Next run will skip tokenization and load from cache |
| 2025-11-22 19:25:36,538 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... |
| 2025-11-22 19:26:10,347 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-22 19:26:10,351 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-22 19:26:10,467 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-22 19:26:11,722 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-22 19:26:19,194 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-22 19:26:20,010 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-22 19:26:20,021 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-22 19:26:20,149 - __main__ - INFO - ✅ Trainer initialized |
| 2025-11-22 19:26:20,150 - __main__ - INFO - |
| 🔬 STEP 1: Baseline Evaluation |
| 2025-11-22 19:26:20,766 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 |
| 2025-11-22 19:26:20,767 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 19:26:20,767 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... |
| 2025-11-22 19:26:20,768 - __main__ - INFO - ====================================================================== |
| 2025-11-22 19:26:42,556 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.0198 |
| 2025-11-22 19:26:42,557 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-22 19:26:42,595 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... |
| 2025-11-22 19:26:42,596 - __main__ - INFO - Batch 1/16... |
| 2025-11-22 19:26:44,238 - __main__ - INFO - Batch 5/16... |
| 2025-11-22 19:26:45,863 - __main__ - INFO - Batch 10/16... |
| 2025-11-22 19:26:47,539 - __main__ - INFO - Batch 15/16... |
| 2025-11-22 19:26:48,147 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-22 19:26:48,160 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-22 19:26:48,168 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-22 19:26:48,169 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-22 19:26:48,171 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision |
| 2025-11-22 19:26:48,172 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-22 19:26:51,610 - __main__ - INFO - ✅ COMET-QE: 0.0383 ± 0.1441 |
| 2025-11-22 19:26:51,614 - __main__ - INFO - |
| ✅ Baseline Results (Test Set): |
| 2025-11-22 19:26:51,615 - __main__ - INFO - BLEU: 0.0198 |
| 2025-11-22 19:26:51,616 - __main__ - INFO - chrF: 17.59 |
| 2025-11-22 19:26:51,616 - __main__ - INFO - COMET: 0.0383 |
| 2025-11-22 19:26:51,617 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 19:26:51,617 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 19:26:51,618 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-22 19:26:51,618 - __main__ - INFO - ====================================================================== |
| 2025-11-22 19:26:51,619 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-22 19:26:51,620 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-22 19:26:52,723 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-22 19:26:52,790 - __main__ - INFO - ✅ Domain chrF: 0.93 |
| 2025-11-22 19:26:52,797 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-22 19:26:52,823 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-22 19:26:52,831 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-22 19:26:52,831 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-22 19:26:52,833 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-22 19:26:55,743 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 |
| 2025-11-22 19:26:55,744 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-22 19:26:55,744 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-22 19:26:55,745 - __main__ - INFO - chrF: 0.93 |
| 2025-11-22 19:26:55,745 - __main__ - INFO - COMET-QE: 0.0000 |
| 2025-11-22 19:26:55,746 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 19:26:56,216 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 |
| 2025-11-22 19:26:56,226 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase1/baseline_results.json |
| 2025-11-22 19:26:56,238 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics |
| 2025-11-22 19:26:56,239 - __main__ - INFO - |
| 🚀 STEP 2: Fine-tuning |
| 2025-11-22 19:26:56,240 - __main__ - INFO - Metrics are being logged to MLflow in real-time |
| 2025-11-22 19:26:56,240 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:26:56,241 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1/logs |
| 2025-11-22 19:26:56,242 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') |
| 2025-11-22 19:26:56,242 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 19:26:56,243 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 19:26:56,244 - __main__ - INFO - 🚀 Starting training... |
| 2025-11-22 19:26:56,244 - __main__ - INFO - ====================================================================== |
| 2025-11-22 19:26:57,393 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) |
| 2025-11-22 19:27:21,616 - __main__ - INFO - 📈 Epoch progress: 0.01/8 (0.1%) |
| 2025-11-22 19:27:45,820 - __main__ - INFO - 📈 Epoch progress: 0.02/8 (0.2%) |
| 2025-11-22 19:28:10,085 - __main__ - INFO - 📈 Epoch progress: 0.03/8 (0.4%) |
| 2025-11-22 19:28:34,780 - __main__ - INFO - 📈 Epoch progress: 0.04/8 (0.5%) |
| 2025-11-22 19:28:59,138 - __main__ - INFO - 📈 Epoch progress: 0.05/8 (0.6%) |
| 2025-11-22 19:29:23,444 - __main__ - INFO - 📈 Epoch progress: 0.06/8 (0.7%) |
| 2025-11-22 19:29:47,282 - __main__ - INFO - 📈 Epoch progress: 0.07/8 (0.8%) |
| 2025-11-22 19:30:10,983 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (0.9%) |
| 2025-11-22 19:30:34,764 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (1.1%) |
| 2025-11-22 19:30:58,732 - __main__ - INFO - 📈 Epoch progress: 0.09/8 (1.2%) |
| 2025-11-22 19:31:22,641 - __main__ - INFO - 📈 Epoch progress: 0.10/8 (1.3%) |
| 2025-11-22 19:31:47,140 - __main__ - INFO - 📈 Epoch progress: 0.11/8 (1.4%) |
| 2025-11-22 19:32:10,785 - __main__ - INFO - 📈 Epoch progress: 0.12/8 (1.5%) |
| 2025-11-22 19:32:34,598 - __main__ - INFO - 📈 Epoch progress: 0.13/8 (1.7%) |
| 2025-11-22 19:32:58,470 - __main__ - INFO - 📈 Epoch progress: 0.14/8 (1.8%) |
| 2025-11-22 19:33:22,516 - __main__ - INFO - 📈 Epoch progress: 0.15/8 (1.9%) |
| 2025-11-22 19:33:46,506 - __main__ - INFO - 📈 Epoch progress: 0.16/8 (2.0%) |
| 2025-11-22 19:34:10,370 - __main__ - INFO - 📈 Epoch progress: 0.17/8 (2.1%) |
| 2025-11-22 19:34:34,008 - __main__ - INFO - 📈 Epoch progress: 0.18/8 (2.2%) |
| 2025-11-22 19:34:57,884 - __main__ - INFO - 📈 Epoch progress: 0.19/8 (2.4%) |
| 2025-11-22 19:54:08,127 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-22 19:54:08,138 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase1.json |
| 2025-11-22 19:54:08,138 - __main__ - INFO - Base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 19:54:08,139 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 19:54:08,142 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-22 19:54:08,143 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-22 19:54:08,144 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 19:54:08,145 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-22 19:54:08,146 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-22 19:54:08,146 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-22 19:54:08,147 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-22 19:54:08,148 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-22 19:54:08,163 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-22 19:54:08,169 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:54:08,905 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-22 19:54:09,890 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251122_195409 |
| 2025-11-22 19:54:09,891 - __main__ - INFO - Run ID: 8b5deaed89ec48e986246e5b801158e3 |
| 2025-11-22 19:54:09,892 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 19:54:10,331 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow |
| 2025-11-22 19:54:10,334 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-22 19:54:10,345 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-22 19:54:10,346 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-22 19:54:10,347 - __main__ - INFO - |
| 📥 Loading base model: Helsinki-NLP/opus-mt-mul-en |
| 2025-11-22 19:54:13,899 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-22 19:54:13,916 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-22 19:54:13,956 - __main__ - INFO - ✅ nllb_swahili_1: 214,915 lines in file |
| 2025-11-22 19:54:14,464 - __main__ - INFO - ✅ nllb_swahili_2: 2,975,542 lines in file |
| 2025-11-22 19:54:14,465 - __main__ - INFO - 📥 Loading nllb_swahili_1 from /tmp/dataset/NLLB.en-sw-1-cleaned_clean.jsonl |
| 2025-11-22 19:54:14,729 - __main__ - INFO - ✅ Loaded 214,915 samples using datasets library (memory-efficient) |
| 2025-11-22 19:54:18,636 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_1 |
| 2025-11-22 19:54:22,160 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_1 |
| 2025-11-22 19:54:22,658 - __main__ - INFO - Loaded 214,915 samples from nllb_swahili_1 (weight: 1.0x) |
| 2025-11-22 19:54:22,663 - __main__ - INFO - 📥 Loading nllb_swahili_2 from /tmp/dataset/NLLB.en-sw-2-cleaned_clean.jsonl |
| 2025-11-22 19:54:24,643 - __main__ - INFO - ✅ Loaded 2,975,542 samples using datasets library (memory-efficient) |
| 2025-11-22 19:54:27,963 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:31,277 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:34,536 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:37,749 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:40,962 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:44,154 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:47,382 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:50,550 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:53,754 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:54:56,995 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:00,239 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:03,493 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:06,729 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:09,937 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:13,139 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:16,411 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:19,744 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:23,053 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:26,373 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:29,693 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:32,966 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:36,255 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:39,548 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:42,824 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:46,080 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:49,373 - __main__ - INFO - Processed 2,600,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:52,653 - __main__ - INFO - Processed 2,700,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:55,930 - __main__ - INFO - Processed 2,800,000 samples from nllb_swahili_2 |
| 2025-11-22 19:55:59,183 - __main__ - INFO - Processed 2,900,000 samples from nllb_swahili_2 |
| 2025-11-22 19:56:01,628 - __main__ - INFO - Loaded 2,975,542 samples from nllb_swahili_2 (weight: 1.0x) |
| 2025-11-22 19:56:01,684 - __main__ - INFO - 📊 Total samples loaded: 3,190,457 |
| 2025-11-22 19:56:01,686 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-22 19:56:01,687 - __main__ - INFO - Sample 1: |
| 2025-11-22 19:56:01,688 - __main__ - INFO - Source (7 words): Taifa wamebadili miungu yao, ingawa siyo miungu?... |
| 2025-11-22 19:56:01,688 - __main__ - INFO - Target (12 words): Has a nation changed its gods, even when they are not gods?... |
| 2025-11-22 19:56:01,690 - __main__ - INFO - Sample 2: |
| 2025-11-22 19:56:01,690 - __main__ - INFO - Source (8 words): Yeye pia atakuwa taifa, naye pia atakuwa mkuu.... |
| 2025-11-22 19:56:01,691 - __main__ - INFO - Target (12 words): He will become a nation also, and he also will be great.... |
| 2025-11-22 19:56:01,692 - __main__ - INFO - Sample 3: |
| 2025-11-22 19:56:01,693 - __main__ - INFO - Source (15 words): Hakika Mimi leo nimewalipa kwa sababu ya kusubiri kwao, bila shaka hao ndio weny... |
| 2025-11-22 19:56:01,693 - __main__ - INFO - Target (17 words): I have rewarded them this Day for their patience |
| 2025-11-22 19:56:05,923 - __main__ - INFO - Filtering statistics: {'kept': 3190457} |
| 2025-11-22 19:56:05,925 - __main__ - INFO - 📊 Samples after filtering: 3,190,457 |
| 2025-11-22 19:56:09,451 - __main__ - INFO - Removed 116 duplicates |
| 2025-11-22 19:56:09,641 - __main__ - INFO - 📊 Samples after deduplication: 3,190,341 |
| 2025-11-22 19:56:15,040 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 19:56:15,041 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-22 19:56:15,041 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 19:56:15,042 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 19:56:15,042 - __main__ - INFO - Test: 159,518 samples |
| 2025-11-22 19:56:15,043 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 19:56:15,413 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-22 19:58:46,518 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-22 19:58:46,523 - __main__ - INFO - Train: 2,711,789 samples |
| 2025-11-22 19:58:46,523 - __main__ - INFO - Validation: 319,034 samples |
| 2025-11-22 19:58:46,524 - __main__ - INFO - Test: 159,518 samples |
|
|
| 2025-11-22 19:58:48,125 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: c82e2605) |
| 2025-11-22 19:58:48,126 - __main__ - INFO - Next run will skip tokenization and load from cache |
| 2025-11-22 19:58:48,126 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=4)... |
| 2025-11-22 19:59:26,483 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-22 19:59:26,485 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-22 19:59:26,617 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-22 19:59:27,759 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-22 19:59:34,104 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-22 19:59:34,612 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-22 19:59:34,618 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-22 19:59:35,142 - __main__ - INFO - 📊 Limiting evaluation dataset from 319046 to 1000 samples |
| 2025-11-22 19:59:35,223 - __main__ - INFO - ✅ Trainer initialized |
| 2025-11-22 19:59:35,230 - __main__ - INFO - |
| 🔬 STEP 1: Baseline Evaluation |
| 2025-11-22 19:59:35,817 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 |
| 2025-11-22 19:59:35,819 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 19:59:35,820 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... |
| 2025-11-22 19:59:35,820 - __main__ - INFO - ====================================================================== |
| 2025-11-22 19:59:57,876 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.0198 |
| 2025-11-22 19:59:57,877 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-22 19:59:57,910 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... |
| 2025-11-22 19:59:57,916 - __main__ - INFO - Batch 1/16... |
| 2025-11-22 19:59:59,612 - __main__ - INFO - Batch 5/16... |
| 2025-11-22 20:00:01,281 - __main__ - INFO - Batch 10/16... |
| 2025-11-22 20:00:02,999 - __main__ - INFO - Batch 15/16... |
| 2025-11-22 20:00:03,636 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-22 20:00:03,653 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-22 20:00:03,662 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-22 20:00:03,663 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-22 20:00:03,664 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision |
| 2025-11-22 20:00:03,666 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-22 20:00:07,052 - __main__ - INFO - ✅ COMET-QE: 0.0383 ± 0.1441 |
| 2025-11-22 20:00:07,052 - __main__ - INFO - |
| ✅ Baseline Results (Test Set): |
| 2025-11-22 20:00:07,053 - __main__ - INFO - BLEU: 0.0198 |
| 2025-11-22 20:00:07,054 - __main__ - INFO - chrF: 17.59 |
| 2025-11-22 20:00:07,055 - __main__ - INFO - COMET: 0.0383 |
| 2025-11-22 20:00:07,055 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 20:00:07,056 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 20:00:07,056 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-22 20:00:07,057 - __main__ - INFO - ====================================================================== |
| 2025-11-22 20:00:07,057 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-22 20:00:07,059 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-22 20:00:08,185 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-22 20:00:08,262 - __main__ - INFO - ✅ Domain chrF: 0.93 |
| 2025-11-22 20:00:08,263 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-22 20:00:08,270 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-22 20:00:08,278 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-22 20:00:08,278 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-22 20:00:08,280 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-22 20:00:10,965 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 |
| 2025-11-22 20:00:10,966 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-22 20:00:10,967 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-22 20:00:10,967 - __main__ - INFO - chrF: 0.93 |
| 2025-11-22 20:00:10,967 - __main__ - INFO - COMET-QE: 0.0000 |
| 2025-11-22 20:00:10,968 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 20:00:11,420 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 |
| 2025-11-22 20:00:11,430 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase1/baseline_results.json |
| 2025-11-22 20:00:11,435 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics |
| 2025-11-22 20:00:11,436 - __main__ - INFO - |
| 🚀 STEP 2: Fine-tuning |
| 2025-11-22 20:00:11,437 - __main__ - INFO - Metrics are being logged to MLflow in real-time |
| 2025-11-22 20:00:11,437 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 20:00:11,438 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1/logs |
| 2025-11-22 20:00:11,439 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') |
| 2025-11-22 20:00:11,440 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 20:00:11,440 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 20:00:11,441 - __main__ - INFO - 🚀 Starting training... |
| 2025-11-22 20:00:11,442 - __main__ - INFO - ====================================================================== |
| 2025-11-22 20:00:13,055 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) |
| 2025-11-22 20:00:38,350 - __main__ - INFO - 📈 Epoch progress: 0.01/8 (0.1%) |
| 2025-11-22 20:01:08,931 - __main__ - INFO - 📈 Epoch progress: 0.02/8 (0.2%) |
| 2025-11-22 20:01:40,702 - __main__ - INFO - 📈 Epoch progress: 0.03/8 (0.4%) |
| 2025-11-22 20:02:07,267 - __main__ - INFO - 📈 Epoch progress: 0.04/8 (0.5%) |
| 2025-11-22 20:02:31,080 - __main__ - INFO - 📈 Epoch progress: 0.05/8 (0.6%) |
| 2025-11-22 20:02:55,120 - __main__ - INFO - 📈 Epoch progress: 0.06/8 (0.7%) |
| 2025-11-22 20:03:18,911 - __main__ - INFO - 📈 Epoch progress: 0.07/8 (0.8%) |
| 2025-11-22 20:03:42,670 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (0.9%) |
| 2025-11-22 20:04:06,358 - __main__ - INFO - 📈 Epoch progress: 0.08/8 (1.1%) |
| 2025-11-22 20:04:30,209 - __main__ - INFO - 📈 Epoch progress: 0.09/8 (1.2%) |
| 2025-11-22 20:04:53,982 - __main__ - INFO - 📈 Epoch progress: 0.10/8 (1.3%) |
| 2025-11-22 20:05:18,464 - __main__ - INFO - 📈 Epoch progress: 0.11/8 (1.4%) |
| 2025-11-22 20:05:42,236 - __main__ - INFO - 📈 Epoch progress: 0.12/8 (1.5%) |
| 2025-11-22 20:06:06,369 - __main__ - INFO - 📈 Epoch progress: 0.13/8 (1.7%) |
| 2025-11-22 20:06:30,324 - __main__ - INFO - 📈 Epoch progress: 0.14/8 (1.8%) |
| 2025-11-22 20:06:54,545 - __main__ - INFO - 📈 Epoch progress: 0.15/8 (1.9%) |
| 2025-11-22 20:07:17,941 - __main__ - INFO - 📈 Epoch progress: 0.16/8 (2.0%) |
| 2025-11-22 20:07:41,589 - __main__ - INFO - 📈 Epoch progress: 0.17/8 (2.1%) |
| 2025-11-22 20:08:05,091 - __main__ - INFO - 📈 Epoch progress: 0.18/8 (2.2%) |
| 2025-11-22 20:08:28,918 - __main__ - INFO - 📈 Epoch progress: 0.19/8 (2.4%) |
| 2025-11-22 20:08:41,181 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1085 |
| 2025-11-22 20:08:41,601 - __main__ - INFO - 📊 Evaluation at step 2000 |
| 2025-11-22 20:08:53,479 - __main__ - INFO - |
| 📤 Pushing checkpoint 2000 to HuggingFace Hub... |
| 2025-11-22 20:08:53,480 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 20:09:03,565 - __main__ - WARNING - ⚠️ Failed to push checkpoint to hub: 404 Client Error. (Request ID: Root=1-6922185f-2864f6c94f85721857318bbc |
|
|
| Repository Not Found for url: https://huggingface.co/api/models/brendaogutu/mul-sw-en-translation-phase1/preupload/main. |
| Please make sure you specified the correct `repo_id` and `repo_type`. |
| If you are trying to access a private or gated repo, make sure you are authenticated. For more details, see https://huggingface.co/docs/huggingface_hub/authentication |
| Note: Creating a commit assumes that the repo already exists on the Huggingface Hub. Please use `create_repo` if it's not the case. |
| 2025-11-22 20:09:27,139 - __main__ - INFO - 📈 Epoch progress: 0.20/8 (2.5%) |
| 2025-11-22 20:09:50,958 - __main__ - INFO - 📈 Epoch progress: 0.21/8 (2.6%) |
| 2025-11-22 20:10:14,586 - __main__ - INFO - 📈 Epoch progress: 0.22/8 (2.7%) |
| 2025-11-22 20:10:38,782 - __main__ - INFO - 📈 Epoch progress: 0.23/8 (2.8%) |
| 2025-11-22 20:11:02,476 - __main__ - INFO - 📈 Epoch progress: 0.24/8 (2.9%) |
| 2025-11-22 20:11:25,551 - __main__ - INFO - 📈 Epoch progress: 0.25/8 (3.1%) |
| 2025-11-22 20:11:49,041 - __main__ - INFO - 📈 Epoch progress: 0.25/8 (3.2%) |
| 2025-11-22 20:12:12,789 - __main__ - INFO - 📈 Epoch progress: 0.26/8 (3.3%) |
| 2025-11-22 20:12:37,301 - __main__ - INFO - 📈 Epoch progress: 0.27/8 (3.4%) |
| 2025-11-22 20:13:01,463 - __main__ - INFO - 📈 Epoch progress: 0.28/8 (3.5%) |
| 2025-11-22 20:13:24,947 - __main__ - INFO - 📈 Epoch progress: 0.29/8 (3.7%) |
| 2025-11-22 20:13:48,819 - __main__ - INFO - 📈 Epoch progress: 0.30/8 (3.8%) |
| 2025-11-22 20:14:12,698 - __main__ - INFO - 📈 Epoch progress: 0.31/8 (3.9%) |
| 2025-11-22 20:14:36,547 - __main__ - INFO - 📈 Epoch progress: 0.32/8 (4.0%) |
| 2025-11-22 20:15:09,789 - __main__ - INFO - 📈 Epoch progress: 0.33/8 (4.1%) |
| 2025-11-22 20:15:33,688 - __main__ - INFO - 📈 Epoch progress: 0.34/8 (4.2%) |
| 2025-11-22 20:15:57,310 - __main__ - INFO - 📈 Epoch progress: 0.35/8 (4.4%) |
| 2025-11-22 20:16:21,091 - __main__ - INFO - 📈 Epoch progress: 0.36/8 (4.5%) |
| 2025-11-22 20:16:44,861 - __main__ - INFO - 📈 Epoch progress: 0.37/8 (4.6%) |
| 2025-11-22 20:17:08,559 - __main__ - INFO - 📈 Epoch progress: 0.38/8 (4.7%) |
| 2025-11-22 20:17:24,279 - __main__ - INFO - 📊 Evaluation 3: BLEU = 0.1403 |
| 2025-11-22 20:17:24,620 - __main__ - INFO - 📊 Evaluation at step 4000 |
| 2025-11-22 20:17:30,144 - __main__ - INFO - |
| 📤 Pushing checkpoint 4000 to HuggingFace Hub... |
| 2025-11-22 20:17:30,145 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 20:17:40,192 - __main__ - INFO - ✅ Checkpoint 4000 pushed successfully! |
| 2025-11-22 20:17:40,193 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-4000 |
| 2025-11-22 20:18:03,497 - __main__ - INFO - 📈 Epoch progress: 0.39/8 (4.8%) |
| 2025-11-22 20:18:27,502 - __main__ - INFO - 📈 Epoch progress: 0.40/8 (5.0%) |
| 2025-11-22 20:18:50,959 - __main__ - INFO - 📈 Epoch progress: 0.41/8 (5.1%) |
| 2025-11-22 20:19:14,786 - __main__ - INFO - 📈 Epoch progress: 0.42/8 (5.2%) |
| 2025-11-22 20:19:38,083 - __main__ - INFO - 📈 Epoch progress: 0.42/8 (5.3%) |
| 2025-11-22 20:20:01,787 - __main__ - INFO - 📈 Epoch progress: 0.43/8 (5.4%) |
| 2025-11-22 20:20:25,229 - __main__ - INFO - 📈 Epoch progress: 0.44/8 (5.5%) |
| 2025-11-22 20:20:48,594 - __main__ - INFO - 📈 Epoch progress: 0.45/8 (5.7%) |
| 2025-11-22 20:21:12,168 - __main__ - INFO - 📈 Epoch progress: 0.46/8 (5.8%) |
| 2025-11-22 20:21:36,348 - __main__ - INFO - 📈 Epoch progress: 0.47/8 (5.9%) |
| 2025-11-22 20:21:59,806 - __main__ - INFO - 📈 Epoch progress: 0.48/8 (6.0%) |
| 2025-11-22 20:22:23,780 - __main__ - INFO - 📈 Epoch progress: 0.49/8 (6.1%) |
| 2025-11-22 20:22:47,397 - __main__ - INFO - 📈 Epoch progress: 0.50/8 (6.3%) |
| 2025-11-22 20:23:11,216 - __main__ - INFO - 📈 Epoch progress: 0.51/8 (6.4%) |
| 2025-11-22 20:23:34,702 - __main__ - INFO - 📈 Epoch progress: 0.52/8 (6.5%) |
| 2025-11-22 20:23:58,147 - __main__ - INFO - 📈 Epoch progress: 0.53/8 (6.6%) |
| 2025-11-22 20:24:21,926 - __main__ - INFO - 📈 Epoch progress: 0.54/8 (6.7%) |
| 2025-11-22 20:24:45,551 - __main__ - INFO - 📈 Epoch progress: 0.55/8 (6.8%) |
| 2025-11-22 20:25:09,313 - __main__ - INFO - 📈 Epoch progress: 0.56/8 (7.0%) |
| 2025-11-22 20:25:33,561 - __main__ - INFO - 📈 Epoch progress: 0.57/8 (7.1%) |
| 2025-11-22 20:25:52,923 - __main__ - INFO - 📊 Evaluation 4: BLEU = 0.1460 |
| 2025-11-22 20:25:53,358 - __main__ - INFO - 📊 Evaluation at step 6000 |
| 2025-11-22 20:25:57,061 - __main__ - INFO - |
| 📤 Pushing checkpoint 6000 to HuggingFace Hub... |
| 2025-11-22 20:25:57,062 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 20:26:07,651 - __main__ - INFO - ✅ Checkpoint 6000 pushed successfully! |
| 2025-11-22 20:26:07,652 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-6000 |
| 2025-11-22 20:26:31,166 - __main__ - INFO - 📈 Epoch progress: 0.58/8 (7.2%) |
| 2025-11-22 20:26:54,743 - __main__ - INFO - 📈 Epoch progress: 0.59/8 (7.3%) |
| 2025-11-22 20:27:18,278 - __main__ - INFO - 📈 Epoch progress: 0.59/8 (7.4%) |
| 2025-11-22 20:27:42,270 - __main__ - INFO - 📈 Epoch progress: 0.60/8 (7.6%) |
| 2025-11-22 20:28:06,898 - __main__ - INFO - 📈 Epoch progress: 0.61/8 (7.7%) |
| 2025-11-22 20:28:30,498 - __main__ - INFO - 📈 Epoch progress: 0.62/8 (7.8%) |
| 2025-11-22 20:28:54,751 - __main__ - INFO - 📈 Epoch progress: 0.63/8 (7.9%) |
| 2025-11-22 20:29:18,490 - __main__ - INFO - 📈 Epoch progress: 0.64/8 (8.0%) |
| 2025-11-22 20:29:41,950 - __main__ - INFO - 📈 Epoch progress: 0.65/8 (8.1%) |
| 2025-11-22 20:30:05,838 - __main__ - INFO - 📈 Epoch progress: 0.66/8 (8.3%) |
| 2025-11-22 20:30:29,703 - __main__ - INFO - 📈 Epoch progress: 0.67/8 (8.4%) |
| 2025-11-22 20:30:53,615 - __main__ - INFO - 📈 Epoch progress: 0.68/8 (8.5%) |
| 2025-11-22 20:31:17,704 - __main__ - INFO - 📈 Epoch progress: 0.69/8 (8.6%) |
| 2025-11-22 20:31:41,537 - __main__ - INFO - 📈 Epoch progress: 0.70/8 (8.7%) |
| 2025-11-22 20:32:05,353 - __main__ - INFO - 📈 Epoch progress: 0.71/8 (8.8%) |
| 2025-11-22 20:32:37,028 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 20:32:37,029 - __main__ - INFO - 📈 Epoch progress: 0.72/8 (9.0%) |
| 2025-11-22 20:32:37,375 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 20:32:49,028 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 20:33:00,709 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 20:33:00,714 - __main__ - INFO - 📈 Epoch progress: 0.73/8 (9.1%) |
| 2025-11-22 20:33:01,151 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 20:33:13,027 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 20:33:24,387 - __main__ - INFO - 📈 Epoch progress: 0.74/8 (9.2%) |
| 2025-11-22 20:33:47,835 - __main__ - INFO - 📈 Epoch progress: 0.75/8 (9.3%) |
| 2025-11-22 20:34:11,110 - __main__ - INFO - 📈 Epoch progress: 0.76/8 (9.4%) |
| 2025-11-22 20:34:26,961 - __main__ - INFO - 📊 Evaluation 5: BLEU = 0.1538 |
| 2025-11-22 20:34:27,322 - __main__ - INFO - 📊 Evaluation at step 8000 |
| 2025-11-22 20:34:33,717 - __main__ - INFO - |
| 📤 Pushing checkpoint 8000 to HuggingFace Hub... |
| 2025-11-22 20:34:33,718 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 20:34:44,612 - __main__ - INFO - ✅ Checkpoint 8000 pushed successfully! |
| 2025-11-22 20:34:44,615 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-8000 |
| 2025-11-22 20:35:07,956 - __main__ - INFO - 📈 Epoch progress: 0.76/8 (9.6%) |
| 2025-11-22 20:35:31,667 - __main__ - INFO - 📈 Epoch progress: 0.77/8 (9.7%) |
| 2025-11-22 20:35:55,140 - __main__ - INFO - 📈 Epoch progress: 0.78/8 (9.8%) |
| 2025-11-22 20:36:18,945 - __main__ - INFO - 📈 Epoch progress: 0.79/8 (9.9%) |
| 2025-11-22 20:36:43,642 - __main__ - INFO - 📈 Epoch progress: 0.80/8 (10.0%) |
| 2025-11-22 20:37:07,303 - __main__ - INFO - 📈 Epoch progress: 0.81/8 (10.1%) |
| 2025-11-22 20:37:31,040 - __main__ - INFO - 📈 Epoch progress: 0.82/8 (10.3%) |
| 2025-11-22 20:37:54,752 - __main__ - INFO - 📈 Epoch progress: 0.83/8 (10.4%) |
| 2025-11-22 20:38:18,185 - __main__ - INFO - 📈 Epoch progress: 0.84/8 (10.5%) |
| 2025-11-22 20:38:41,713 - __main__ - INFO - 📈 Epoch progress: 0.85/8 (10.6%) |
| 2025-11-22 20:39:05,292 - __main__ - INFO - 📈 Epoch progress: 0.86/8 (10.7%) |
| 2025-11-22 20:39:28,872 - __main__ - INFO - 📈 Epoch progress: 0.87/8 (10.9%) |
| 2025-11-22 20:39:52,309 - __main__ - INFO - 📈 Epoch progress: 0.88/8 (11.0%) |
| 2025-11-22 20:40:16,091 - __main__ - INFO - 📈 Epoch progress: 0.89/8 (11.1%) |
| 2025-11-22 20:40:39,485 - __main__ - INFO - 📈 Epoch progress: 0.90/8 (11.2%) |
| 2025-11-22 20:41:03,175 - __main__ - INFO - 📈 Epoch progress: 0.91/8 (11.3%) |
| 2025-11-22 20:41:26,580 - __main__ - INFO - 📈 Epoch progress: 0.92/8 (11.4%) |
| 2025-11-22 20:41:50,347 - __main__ - INFO - 📈 Epoch progress: 0.93/8 (11.6%) |
| 2025-11-22 20:42:13,732 - __main__ - INFO - 📈 Epoch progress: 0.93/8 (11.7%) |
| 2025-11-22 20:42:37,615 - __main__ - INFO - 📈 Epoch progress: 0.94/8 (11.8%) |
| 2025-11-22 20:42:52,788 - __main__ - INFO - 📊 Evaluation 6: BLEU = 0.1541 |
| 2025-11-22 20:42:53,140 - __main__ - INFO - 📊 Evaluation at step 10000 |
| 2025-11-22 20:42:57,687 - __main__ - INFO - |
| 📤 Pushing checkpoint 10000 to HuggingFace Hub... |
| 2025-11-22 20:42:57,688 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 20:43:08,616 - __main__ - INFO - ✅ Checkpoint 10000 pushed successfully! |
| 2025-11-22 20:43:08,619 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-10000 |
| 2025-11-22 20:43:31,749 - __main__ - INFO - 📈 Epoch progress: 0.95/8 (11.9%) |
| 2025-11-22 20:43:55,608 - __main__ - INFO - 📈 Epoch progress: 0.96/8 (12.0%) |
| 2025-11-22 20:44:19,143 - __main__ - INFO - 📈 Epoch progress: 0.97/8 (12.2%) |
| 2025-11-22 20:44:42,575 - __main__ - INFO - 📈 Epoch progress: 0.98/8 (12.3%) |
| 2025-11-22 20:45:06,633 - __main__ - INFO - 📈 Epoch progress: 0.99/8 (12.4%) |
| 2025-11-22 20:45:30,901 - __main__ - INFO - 📈 Epoch progress: 1.00/8 (12.5%) |
| 2025-11-22 20:45:54,569 - __main__ - INFO - 📈 Epoch progress: 1.01/8 (12.6%) |
| 2025-11-22 20:46:18,121 - __main__ - INFO - 📈 Epoch progress: 1.02/8 (12.7%) |
| 2025-11-22 20:46:41,888 - __main__ - INFO - 📈 Epoch progress: 1.03/8 (12.9%) |
| 2025-11-22 20:47:06,029 - __main__ - INFO - 📈 Epoch progress: 1.04/8 (13.0%) |
| 2025-11-22 20:47:29,508 - __main__ - INFO - 📈 Epoch progress: 1.05/8 (13.1%) |
| 2025-11-22 20:47:52,465 - __main__ - INFO - 📈 Epoch progress: 1.06/8 (13.2%) |
| 2025-11-22 20:48:15,982 - __main__ - INFO - 📈 Epoch progress: 1.07/8 (13.3%) |
| 2025-11-22 20:48:39,691 - __main__ - INFO - 📈 Epoch progress: 1.08/8 (13.5%) |
| 2025-11-22 20:49:03,378 - __main__ - INFO - 📈 Epoch progress: 1.09/8 (13.6%) |
| 2025-11-22 20:49:26,543 - __main__ - INFO - 📈 Epoch progress: 1.09/8 (13.7%) |
| 2025-11-22 20:49:49,871 - __main__ - INFO - 📈 Epoch progress: 1.10/8 (13.8%) |
| 2025-11-22 20:50:13,500 - __main__ - INFO - 📈 Epoch progress: 1.11/8 (13.9%) |
| 2025-11-22 20:50:37,515 - __main__ - INFO - 📈 Epoch progress: 1.12/8 (14.0%) |
| 2025-11-22 20:51:01,136 - __main__ - INFO - 📈 Epoch progress: 1.13/8 (14.2%) |
| 2025-11-22 20:51:25,040 - __main__ - INFO - 📊 Evaluation 7: BLEU = 0.1463 |
| 2025-11-22 20:51:25,397 - __main__ - INFO - 📊 Evaluation at step 12000 |
| 2025-11-22 20:51:30,017 - __main__ - INFO - |
| 📤 Pushing checkpoint 12000 to HuggingFace Hub... |
| 2025-11-22 20:51:30,018 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 20:51:39,904 - __main__ - INFO - ✅ Checkpoint 12000 pushed successfully! |
| 2025-11-22 20:51:39,914 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-12000 |
| 2025-11-22 20:52:03,513 - __main__ - INFO - 📈 Epoch progress: 1.14/8 (14.3%) |
| 2025-11-22 20:52:27,285 - __main__ - INFO - 📈 Epoch progress: 1.15/8 (14.4%) |
| 2025-11-22 20:52:50,714 - __main__ - INFO - 📈 Epoch progress: 1.16/8 (14.5%) |
| 2025-11-22 20:53:14,707 - __main__ - INFO - 📈 Epoch progress: 1.17/8 (14.6%) |
| 2025-11-22 20:53:38,315 - __main__ - INFO - 📈 Epoch progress: 1.18/8 (14.7%) |
| 2025-11-22 20:54:02,170 - __main__ - INFO - 📈 Epoch progress: 1.19/8 (14.9%) |
| 2025-11-22 20:54:25,745 - __main__ - INFO - 📈 Epoch progress: 1.20/8 (15.0%) |
| 2025-11-22 20:54:56,587 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 20:54:56,589 - __main__ - INFO - 📈 Epoch progress: 1.21/8 (15.1%) |
| 2025-11-22 20:54:56,904 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 20:55:20,217 - __main__ - INFO - 📈 Epoch progress: 1.22/8 (15.2%) |
| 2025-11-22 20:55:43,949 - __main__ - INFO - 📈 Epoch progress: 1.23/8 (15.3%) |
| 2025-11-22 20:56:07,794 - __main__ - INFO - 📈 Epoch progress: 1.24/8 (15.5%) |
| 2025-11-22 20:56:31,239 - __main__ - INFO - 📈 Epoch progress: 1.25/8 (15.6%) |
| 2025-11-22 20:56:54,645 - __main__ - INFO - 📈 Epoch progress: 1.26/8 (15.7%) |
| 2025-11-22 20:57:17,680 - __main__ - INFO - 📈 Epoch progress: 1.26/8 (15.8%) |
| 2025-11-22 20:57:41,619 - __main__ - INFO - 📈 Epoch progress: 1.27/8 (15.9%) |
| 2025-11-22 20:58:05,294 - __main__ - INFO - 📈 Epoch progress: 1.28/8 (16.0%) |
| 2025-11-22 20:58:28,616 - __main__ - INFO - 📈 Epoch progress: 1.29/8 (16.2%) |
| 2025-11-22 20:58:52,621 - __main__ - INFO - 📈 Epoch progress: 1.30/8 (16.3%) |
| 2025-11-22 20:59:16,154 - __main__ - INFO - 📈 Epoch progress: 1.31/8 (16.4%) |
| 2025-11-22 20:59:39,666 - __main__ - INFO - 📈 Epoch progress: 1.32/8 (16.5%) |
| 2025-11-22 20:59:56,828 - __main__ - INFO - 📊 Evaluation 8: BLEU = 0.1585 |
| 2025-11-22 20:59:57,166 - __main__ - INFO - 📊 Evaluation at step 14000 |
| 2025-11-22 21:00:00,516 - __main__ - INFO - |
| 📤 Pushing checkpoint 14000 to HuggingFace Hub... |
| 2025-11-22 21:00:00,517 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 21:00:12,536 - __main__ - INFO - ✅ Checkpoint 14000 pushed successfully! |
| 2025-11-22 21:00:12,538 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-14000 |
| 2025-11-22 21:00:36,120 - __main__ - INFO - 📈 Epoch progress: 1.33/8 (16.6%) |
| 2025-11-22 21:00:59,239 - __main__ - INFO - 📈 Epoch progress: 1.34/8 (16.8%) |
| 2025-11-22 21:01:23,166 - __main__ - INFO - 📈 Epoch progress: 1.35/8 (16.9%) |
| 2025-11-22 21:01:46,902 - __main__ - INFO - 📈 Epoch progress: 1.36/8 (17.0%) |
| 2025-11-22 21:02:10,577 - __main__ - INFO - 📈 Epoch progress: 1.37/8 (17.1%) |
| 2025-11-22 21:02:36,636 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:02:36,638 - __main__ - INFO - 📈 Epoch progress: 1.38/8 (17.2%) |
| 2025-11-22 21:02:36,940 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:03:00,616 - __main__ - INFO - 📈 Epoch progress: 1.39/8 (17.3%) |
| 2025-11-22 21:03:24,289 - __main__ - INFO - 📈 Epoch progress: 1.40/8 (17.5%) |
| 2025-11-22 21:03:47,887 - __main__ - INFO - 📈 Epoch progress: 1.41/8 (17.6%) |
| 2025-11-22 21:04:11,815 - __main__ - INFO - 📈 Epoch progress: 1.42/8 (17.7%) |
| 2025-11-22 21:04:35,610 - __main__ - INFO - 📈 Epoch progress: 1.43/8 (17.8%) |
| 2025-11-22 21:04:59,042 - __main__ - INFO - 📈 Epoch progress: 1.43/8 (17.9%) |
| 2025-11-22 21:05:22,497 - __main__ - INFO - 📈 Epoch progress: 1.44/8 (18.1%) |
| 2025-11-22 21:05:45,903 - __main__ - INFO - 📈 Epoch progress: 1.45/8 (18.2%) |
| 2025-11-22 21:06:09,033 - __main__ - INFO - 📈 Epoch progress: 1.46/8 (18.3%) |
| 2025-11-22 21:06:32,773 - __main__ - INFO - 📈 Epoch progress: 1.47/8 (18.4%) |
| 2025-11-22 21:06:56,585 - __main__ - INFO - 📈 Epoch progress: 1.48/8 (18.5%) |
| 2025-11-22 21:07:20,075 - __main__ - INFO - 📈 Epoch progress: 1.49/8 (18.6%) |
| 2025-11-22 21:07:43,689 - __main__ - INFO - 📈 Epoch progress: 1.50/8 (18.8%) |
| 2025-11-22 21:08:07,130 - __main__ - INFO - 📈 Epoch progress: 1.51/8 (18.9%) |
| 2025-11-22 21:08:22,384 - __main__ - INFO - 📊 Evaluation 9: BLEU = 0.1666 |
| 2025-11-22 21:08:22,727 - __main__ - INFO - 📊 Evaluation at step 16000 |
| 2025-11-22 21:08:27,275 - __main__ - INFO - |
| 📤 Pushing checkpoint 16000 to HuggingFace Hub... |
| 2025-11-22 21:08:27,276 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 21:08:43,289 - __main__ - INFO - ✅ Checkpoint 16000 pushed successfully! |
| 2025-11-22 21:08:43,290 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-16000 |
| 2025-11-22 21:09:06,478 - __main__ - INFO - 📈 Epoch progress: 1.52/8 (19.0%) |
| 2025-11-22 21:09:30,626 - __main__ - INFO - 📈 Epoch progress: 1.53/8 (19.1%) |
| 2025-11-22 21:09:54,017 - __main__ - INFO - 📈 Epoch progress: 1.54/8 (19.2%) |
| 2025-11-22 21:10:17,768 - __main__ - INFO - 📈 Epoch progress: 1.55/8 (19.4%) |
| 2025-11-22 21:10:41,731 - __main__ - INFO - 📈 Epoch progress: 1.56/8 (19.5%) |
| 2025-11-22 21:11:05,567 - __main__ - INFO - 📈 Epoch progress: 1.57/8 (19.6%) |
| 2025-11-22 21:11:29,040 - __main__ - INFO - 📈 Epoch progress: 1.58/8 (19.7%) |
| 2025-11-22 21:11:53,073 - __main__ - INFO - 📈 Epoch progress: 1.59/8 (19.8%) |
| 2025-11-22 21:12:17,169 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:12:17,177 - __main__ - INFO - 📈 Epoch progress: 1.60/8 (19.9%) |
| 2025-11-22 21:12:17,481 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:12:29,345 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:12:41,169 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:12:41,170 - __main__ - INFO - 📈 Epoch progress: 1.60/8 (20.1%) |
| 2025-11-22 21:12:41,465 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:13:04,807 - __main__ - INFO - 📈 Epoch progress: 1.61/8 (20.2%) |
| 2025-11-22 21:13:28,403 - __main__ - INFO - 📈 Epoch progress: 1.62/8 (20.3%) |
| 2025-11-22 21:13:52,653 - __main__ - INFO - 📈 Epoch progress: 1.63/8 (20.4%) |
| 2025-11-22 21:14:15,499 - __main__ - INFO - 📈 Epoch progress: 1.64/8 (20.5%) |
| 2025-11-22 21:14:40,686 - __main__ - INFO - 📈 Epoch progress: 1.65/8 (20.6%) |
| 2025-11-22 21:15:05,650 - __main__ - INFO - 📈 Epoch progress: 1.66/8 (20.8%) |
| 2025-11-22 21:15:29,482 - __main__ - INFO - 📈 Epoch progress: 1.67/8 (20.9%) |
| 2025-11-22 21:15:52,896 - __main__ - INFO - 📈 Epoch progress: 1.68/8 (21.0%) |
| 2025-11-22 21:16:16,430 - __main__ - INFO - 📈 Epoch progress: 1.69/8 (21.1%) |
| 2025-11-22 21:16:40,428 - __main__ - INFO - 📈 Epoch progress: 1.70/8 (21.2%) |
| 2025-11-22 21:16:58,183 - __main__ - INFO - 📊 Evaluation 10: BLEU = 0.1674 |
| 2025-11-22 21:16:58,533 - __main__ - INFO - 📊 Evaluation at step 18000 |
| 2025-11-22 21:17:07,291 - __main__ - INFO - |
| 📤 Pushing checkpoint 18000 to HuggingFace Hub... |
| 2025-11-22 21:17:07,292 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 21:17:24,438 - __main__ - INFO - ✅ Checkpoint 18000 pushed successfully! |
| 2025-11-22 21:17:24,439 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-18000 |
| 2025-11-22 21:17:47,446 - __main__ - INFO - 📈 Epoch progress: 1.71/8 (21.4%) |
| 2025-11-22 21:18:11,052 - __main__ - INFO - 📈 Epoch progress: 1.72/8 (21.5%) |
| 2025-11-22 21:18:35,198 - __main__ - INFO - 📈 Epoch progress: 1.73/8 (21.6%) |
| 2025-11-22 21:18:59,250 - __main__ - INFO - 📈 Epoch progress: 1.74/8 (21.7%) |
| 2025-11-22 21:19:22,877 - __main__ - INFO - 📈 Epoch progress: 1.75/8 (21.8%) |
| 2025-11-22 21:19:46,356 - __main__ - INFO - 📈 Epoch progress: 1.76/8 (21.9%) |
| 2025-11-22 21:20:09,961 - __main__ - INFO - 📈 Epoch progress: 1.77/8 (22.1%) |
| 2025-11-22 21:20:34,266 - __main__ - INFO - 📈 Epoch progress: 1.77/8 (22.2%) |
| 2025-11-22 21:20:58,401 - __main__ - INFO - 📈 Epoch progress: 1.78/8 (22.3%) |
| 2025-11-22 21:21:23,028 - __main__ - INFO - 📈 Epoch progress: 1.79/8 (22.4%) |
| 2025-11-22 21:21:46,582 - __main__ - INFO - 📈 Epoch progress: 1.80/8 (22.5%) |
| 2025-11-22 21:22:10,298 - __main__ - INFO - 📈 Epoch progress: 1.81/8 (22.7%) |
| 2025-11-22 21:22:38,997 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:22:38,998 - __main__ - INFO - 📈 Epoch progress: 1.82/8 (22.8%) |
| 2025-11-22 21:22:39,347 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:22:51,039 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:23:02,730 - __main__ - INFO - 📈 Epoch progress: 1.83/8 (22.9%) |
| 2025-11-22 21:23:25,793 - __main__ - INFO - 📈 Epoch progress: 1.84/8 (23.0%) |
| 2025-11-22 21:23:49,391 - __main__ - INFO - 📈 Epoch progress: 1.85/8 (23.1%) |
| 2025-11-22 21:24:13,588 - __main__ - INFO - 📈 Epoch progress: 1.86/8 (23.2%) |
| 2025-11-22 21:24:37,046 - __main__ - INFO - 📈 Epoch progress: 1.87/8 (23.4%) |
| 2025-11-22 21:25:00,995 - __main__ - INFO - 📈 Epoch progress: 1.88/8 (23.5%) |
| 2025-11-22 21:25:24,480 - __main__ - INFO - 📈 Epoch progress: 1.89/8 (23.6%) |
| 2025-11-22 21:25:46,690 - __main__ - INFO - 📊 Evaluation 11: BLEU = 0.1619 |
| 2025-11-22 21:25:47,056 - __main__ - INFO - 📊 Evaluation at step 20000 |
| 2025-11-22 21:25:53,323 - __main__ - INFO - |
| 📤 Pushing checkpoint 20000 to HuggingFace Hub... |
| 2025-11-22 21:25:53,324 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 21:26:05,726 - __main__ - INFO - ✅ Checkpoint 20000 pushed successfully! |
| 2025-11-22 21:26:05,728 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-20000 |
| 2025-11-22 21:26:29,812 - __main__ - INFO - 📈 Epoch progress: 1.90/8 (23.7%) |
| 2025-11-22 21:26:53,847 - __main__ - INFO - 📈 Epoch progress: 1.91/8 (23.8%) |
| 2025-11-22 21:27:17,422 - __main__ - INFO - 📈 Epoch progress: 1.92/8 (24.0%) |
| 2025-11-22 21:27:42,006 - __main__ - INFO - 📈 Epoch progress: 1.93/8 (24.1%) |
| 2025-11-22 21:28:05,446 - __main__ - INFO - 📈 Epoch progress: 1.94/8 (24.2%) |
| 2025-11-22 21:28:29,168 - __main__ - INFO - 📈 Epoch progress: 1.94/8 (24.3%) |
| 2025-11-22 21:28:52,701 - __main__ - INFO - 📈 Epoch progress: 1.95/8 (24.4%) |
| 2025-11-22 21:29:16,950 - __main__ - INFO - 📈 Epoch progress: 1.96/8 (24.5%) |
| 2025-11-22 21:29:40,772 - __main__ - INFO - 📈 Epoch progress: 1.97/8 (24.7%) |
| 2025-11-22 21:30:04,317 - __main__ - INFO - 📈 Epoch progress: 1.98/8 (24.8%) |
| 2025-11-22 21:30:27,826 - __main__ - INFO - 📈 Epoch progress: 1.99/8 (24.9%) |
| 2025-11-22 21:30:52,170 - __main__ - INFO - 📈 Epoch progress: 2.00/8 (25.0%) |
| 2025-11-22 21:31:15,774 - __main__ - INFO - 📈 Epoch progress: 2.01/8 (25.1%) |
| 2025-11-22 21:31:39,640 - __main__ - INFO - 📈 Epoch progress: 2.02/8 (25.3%) |
| 2025-11-22 21:32:02,847 - __main__ - INFO - 📈 Epoch progress: 2.03/8 (25.4%) |
| 2025-11-22 21:32:27,451 - __main__ - INFO - 📈 Epoch progress: 2.04/8 (25.5%) |
| 2025-11-22 21:32:51,684 - __main__ - INFO - 📈 Epoch progress: 2.05/8 (25.6%) |
| 2025-11-22 21:33:15,308 - __main__ - INFO - 📈 Epoch progress: 2.06/8 (25.7%) |
| 2025-11-22 21:33:38,738 - __main__ - INFO - 📈 Epoch progress: 2.07/8 (25.8%) |
| 2025-11-22 21:34:02,226 - __main__ - INFO - 📈 Epoch progress: 2.08/8 (26.0%) |
| 2025-11-22 21:34:20,000 - __main__ - INFO - 📊 Evaluation 12: BLEU = 0.1677 |
| 2025-11-22 21:34:20,369 - __main__ - INFO - 📊 Evaluation at step 22000 |
| 2025-11-22 21:34:24,871 - __main__ - INFO - |
| 📤 Pushing checkpoint 22000 to HuggingFace Hub... |
| 2025-11-22 21:34:24,872 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 21:34:34,963 - __main__ - INFO - ✅ Checkpoint 22000 pushed successfully! |
| 2025-11-22 21:34:34,965 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-22000 |
| 2025-11-22 21:34:58,374 - __main__ - INFO - 📈 Epoch progress: 2.09/8 (26.1%) |
| 2025-11-22 21:35:22,174 - __main__ - INFO - 📈 Epoch progress: 2.10/8 (26.2%) |
| 2025-11-22 21:35:45,215 - __main__ - INFO - 📈 Epoch progress: 2.10/8 (26.3%) |
| 2025-11-22 21:36:08,584 - __main__ - INFO - 📈 Epoch progress: 2.11/8 (26.4%) |
| 2025-11-22 21:36:32,144 - __main__ - INFO - 📈 Epoch progress: 2.12/8 (26.5%) |
| 2025-11-22 21:36:56,099 - __main__ - INFO - 📈 Epoch progress: 2.13/8 (26.7%) |
| 2025-11-22 21:37:19,843 - __main__ - INFO - 📈 Epoch progress: 2.14/8 (26.8%) |
| 2025-11-22 21:37:43,099 - __main__ - INFO - 📈 Epoch progress: 2.15/8 (26.9%) |
| 2025-11-22 21:38:07,047 - __main__ - INFO - 📈 Epoch progress: 2.16/8 (27.0%) |
| 2025-11-22 21:38:30,112 - __main__ - INFO - 📈 Epoch progress: 2.17/8 (27.1%) |
| 2025-11-22 21:38:53,928 - __main__ - INFO - 📈 Epoch progress: 2.18/8 (27.3%) |
| 2025-11-22 21:39:18,208 - __main__ - INFO - 📈 Epoch progress: 2.19/8 (27.4%) |
| 2025-11-22 21:39:42,762 - __main__ - INFO - 📈 Epoch progress: 2.20/8 (27.5%) |
| 2025-11-22 21:40:06,488 - __main__ - INFO - 📈 Epoch progress: 2.21/8 (27.6%) |
| 2025-11-22 21:40:30,471 - __main__ - INFO - 📈 Epoch progress: 2.22/8 (27.7%) |
| 2025-11-22 21:40:53,743 - __main__ - INFO - 📈 Epoch progress: 2.23/8 (27.8%) |
| 2025-11-22 21:41:17,384 - __main__ - INFO - 📈 Epoch progress: 2.24/8 (28.0%) |
| 2025-11-22 21:41:40,963 - __main__ - INFO - 📈 Epoch progress: 2.25/8 (28.1%) |
| 2025-11-22 21:42:05,101 - __main__ - INFO - 📈 Epoch progress: 2.26/8 (28.2%) |
| 2025-11-22 21:42:28,584 - __main__ - INFO - 📈 Epoch progress: 2.27/8 (28.3%) |
| 2025-11-22 21:42:49,035 - __main__ - INFO - 📊 Evaluation 13: BLEU = 0.1630 |
| 2025-11-22 21:42:49,388 - __main__ - INFO - 📊 Evaluation at step 24000 |
| 2025-11-22 21:42:54,416 - __main__ - INFO - |
| 📤 Pushing checkpoint 24000 to HuggingFace Hub... |
| 2025-11-22 21:42:54,417 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 21:43:11,101 - __main__ - INFO - ✅ Checkpoint 24000 pushed successfully! |
| 2025-11-22 21:43:11,114 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-24000 |
| 2025-11-22 21:43:34,133 - __main__ - INFO - 📈 Epoch progress: 2.27/8 (28.4%) |
| 2025-11-22 21:43:58,647 - __main__ - INFO - 📈 Epoch progress: 2.28/8 (28.6%) |
| 2025-11-22 21:44:22,526 - __main__ - INFO - 📈 Epoch progress: 2.29/8 (28.7%) |
| 2025-11-22 21:44:56,326 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:44:56,328 - __main__ - INFO - 📈 Epoch progress: 2.30/8 (28.8%) |
| 2025-11-22 21:44:56,642 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 21:45:20,598 - __main__ - INFO - 📈 Epoch progress: 2.31/8 (28.9%) |
| 2025-11-22 21:45:44,794 - __main__ - INFO - 📈 Epoch progress: 2.32/8 (29.0%) |
| 2025-11-22 21:46:08,169 - __main__ - INFO - 📈 Epoch progress: 2.33/8 (29.1%) |
| 2025-11-22 21:46:31,960 - __main__ - INFO - 📈 Epoch progress: 2.34/8 (29.3%) |
| 2025-11-22 21:46:55,735 - __main__ - INFO - 📈 Epoch progress: 2.35/8 (29.4%) |
| 2025-11-22 21:47:19,450 - __main__ - INFO - 📈 Epoch progress: 2.36/8 (29.5%) |
| 2025-11-22 21:47:42,862 - __main__ - INFO - 📈 Epoch progress: 2.37/8 (29.6%) |
| 2025-11-22 21:48:07,254 - __main__ - INFO - 📈 Epoch progress: 2.38/8 (29.7%) |
| 2025-11-22 21:48:31,170 - __main__ - INFO - 📈 Epoch progress: 2.39/8 (29.9%) |
| 2025-11-22 21:48:55,953 - __main__ - INFO - 📈 Epoch progress: 2.40/8 (30.0%) |
| 2025-11-22 21:49:19,508 - __main__ - INFO - 📈 Epoch progress: 2.41/8 (30.1%) |
| 2025-11-22 21:49:43,336 - __main__ - INFO - 📈 Epoch progress: 2.42/8 (30.2%) |
| 2025-11-22 21:50:07,127 - __main__ - INFO - 📈 Epoch progress: 2.43/8 (30.3%) |
| 2025-11-22 21:50:31,261 - __main__ - INFO - 📈 Epoch progress: 2.44/8 (30.4%) |
| 2025-11-22 21:50:55,006 - __main__ - INFO - 📈 Epoch progress: 2.44/8 (30.6%) |
| 2025-11-22 21:51:18,988 - __main__ - INFO - 📈 Epoch progress: 2.45/8 (30.7%) |
| 2025-11-22 21:51:41,415 - __main__ - INFO - 📊 Evaluation 14: BLEU = 0.1721 |
| 2025-11-22 21:51:41,771 - __main__ - INFO - 📊 Evaluation at step 26000 |
| 2025-11-22 21:51:47,075 - __main__ - INFO - |
| 📤 Pushing checkpoint 26000 to HuggingFace Hub... |
| 2025-11-22 21:51:47,076 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 21:51:58,317 - __main__ - INFO - ✅ Checkpoint 26000 pushed successfully! |
| 2025-11-22 21:51:58,323 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-26000 |
| 2025-11-22 21:52:21,800 - __main__ - INFO - 📈 Epoch progress: 2.46/8 (30.8%) |
| 2025-11-22 21:52:45,848 - __main__ - INFO - 📈 Epoch progress: 2.47/8 (30.9%) |
| 2025-11-22 21:53:09,666 - __main__ - INFO - 📈 Epoch progress: 2.48/8 (31.0%) |
| 2025-11-22 21:53:33,443 - __main__ - INFO - 📈 Epoch progress: 2.49/8 (31.1%) |
| 2025-11-22 21:53:56,636 - __main__ - INFO - 📈 Epoch progress: 2.50/8 (31.3%) |
| 2025-11-22 21:54:20,060 - __main__ - INFO - 📈 Epoch progress: 2.51/8 (31.4%) |
| 2025-11-22 21:54:44,973 - __main__ - INFO - 📈 Epoch progress: 2.52/8 (31.5%) |
| 2025-11-22 21:55:08,222 - __main__ - INFO - 📈 Epoch progress: 2.53/8 (31.6%) |
| 2025-11-22 21:55:31,852 - __main__ - INFO - 📈 Epoch progress: 2.54/8 (31.7%) |
| 2025-11-22 21:55:55,256 - __main__ - INFO - 📈 Epoch progress: 2.55/8 (31.9%) |
| 2025-11-22 21:56:18,899 - __main__ - INFO - 📈 Epoch progress: 2.56/8 (32.0%) |
| 2025-11-22 21:56:42,547 - __main__ - INFO - 📈 Epoch progress: 2.57/8 (32.1%) |
| 2025-11-22 21:57:06,146 - __main__ - INFO - 📈 Epoch progress: 2.58/8 (32.2%) |
| 2025-11-22 21:57:30,165 - __main__ - INFO - 📈 Epoch progress: 2.59/8 (32.3%) |
| 2025-11-22 21:57:53,581 - __main__ - INFO - 📈 Epoch progress: 2.60/8 (32.4%) |
| 2025-11-22 21:58:17,968 - __main__ - INFO - 📈 Epoch progress: 2.61/8 (32.6%) |
| 2025-11-22 21:58:41,705 - __main__ - INFO - 📈 Epoch progress: 2.61/8 (32.7%) |
| 2025-11-22 21:59:05,247 - __main__ - INFO - 📈 Epoch progress: 2.62/8 (32.8%) |
| 2025-11-22 21:59:28,518 - __main__ - INFO - 📈 Epoch progress: 2.63/8 (32.9%) |
| 2025-11-22 21:59:52,385 - __main__ - INFO - 📈 Epoch progress: 2.64/8 (33.0%) |
| 2025-11-22 22:00:16,924 - __main__ - INFO - 📊 Evaluation 15: BLEU = 0.1706 |
| 2025-11-22 22:00:17,285 - __main__ - INFO - 📊 Evaluation at step 28000 |
| 2025-11-22 22:00:22,699 - __main__ - INFO - |
| 📤 Pushing checkpoint 28000 to HuggingFace Hub... |
| 2025-11-22 22:00:22,718 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 22:00:34,097 - __main__ - INFO - ✅ Checkpoint 28000 pushed successfully! |
| 2025-11-22 22:00:34,099 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-28000 |
| 2025-11-22 22:00:57,572 - __main__ - INFO - 📈 Epoch progress: 2.65/8 (33.2%) |
| 2025-11-22 22:01:21,910 - __main__ - INFO - 📈 Epoch progress: 2.66/8 (33.3%) |
| 2025-11-22 22:01:45,653 - __main__ - INFO - 📈 Epoch progress: 2.67/8 (33.4%) |
| 2025-11-22 22:02:09,324 - __main__ - INFO - 📈 Epoch progress: 2.68/8 (33.5%) |
| 2025-11-22 22:02:32,388 - __main__ - INFO - 📈 Epoch progress: 2.69/8 (33.6%) |
| 2025-11-22 22:02:55,915 - __main__ - INFO - 📈 Epoch progress: 2.70/8 (33.7%) |
| 2025-11-22 22:03:19,709 - __main__ - INFO - 📈 Epoch progress: 2.71/8 (33.9%) |
| 2025-11-22 22:03:42,372 - __main__ - INFO - 📈 Epoch progress: 2.72/8 (34.0%) |
| 2025-11-22 22:04:14,921 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:04:14,922 - __main__ - INFO - 📈 Epoch progress: 2.73/8 (34.1%) |
| 2025-11-22 22:04:15,222 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:04:39,431 - __main__ - INFO - 📈 Epoch progress: 2.74/8 (34.2%) |
| 2025-11-22 22:05:03,047 - __main__ - INFO - 📈 Epoch progress: 2.75/8 (34.3%) |
| 2025-11-22 22:05:26,149 - __main__ - INFO - 📈 Epoch progress: 2.76/8 (34.5%) |
| 2025-11-22 22:05:49,543 - __main__ - INFO - 📈 Epoch progress: 2.77/8 (34.6%) |
| 2025-11-22 22:06:13,007 - __main__ - INFO - 📈 Epoch progress: 2.78/8 (34.7%) |
| 2025-11-22 22:06:36,910 - __main__ - INFO - 📈 Epoch progress: 2.78/8 (34.8%) |
| 2025-11-22 22:07:00,093 - __main__ - INFO - 📈 Epoch progress: 2.79/8 (34.9%) |
| 2025-11-22 22:07:23,534 - __main__ - INFO - 📈 Epoch progress: 2.80/8 (35.0%) |
| 2025-11-22 22:07:47,807 - __main__ - INFO - 📈 Epoch progress: 2.81/8 (35.2%) |
| 2025-11-22 22:08:11,112 - __main__ - INFO - 📈 Epoch progress: 2.82/8 (35.3%) |
| 2025-11-22 22:08:34,882 - __main__ - INFO - 📈 Epoch progress: 2.83/8 (35.4%) |
| 2025-11-22 22:08:54,341 - __main__ - INFO - 📊 Evaluation 16: BLEU = 0.1744 |
| 2025-11-22 22:08:54,690 - __main__ - INFO - 📊 Evaluation at step 30000 |
| 2025-11-22 22:08:58,267 - __main__ - INFO - |
| 📤 Pushing checkpoint 30000 to HuggingFace Hub... |
| 2025-11-22 22:08:58,267 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 22:09:10,043 - __main__ - INFO - ✅ Checkpoint 30000 pushed successfully! |
| 2025-11-22 22:09:10,045 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-30000 |
| 2025-11-22 22:09:33,448 - __main__ - INFO - 📈 Epoch progress: 2.84/8 (35.5%) |
| 2025-11-22 22:09:57,022 - __main__ - INFO - 📈 Epoch progress: 2.85/8 (35.6%) |
| 2025-11-22 22:10:20,696 - __main__ - INFO - 📈 Epoch progress: 2.86/8 (35.8%) |
| 2025-11-22 22:10:44,433 - __main__ - INFO - 📈 Epoch progress: 2.87/8 (35.9%) |
| 2025-11-22 22:11:08,015 - __main__ - INFO - 📈 Epoch progress: 2.88/8 (36.0%) |
| 2025-11-22 22:11:31,148 - __main__ - INFO - 📈 Epoch progress: 2.89/8 (36.1%) |
| 2025-11-22 22:11:55,004 - __main__ - INFO - 📈 Epoch progress: 2.90/8 (36.2%) |
| 2025-11-22 22:12:18,689 - __main__ - INFO - 📈 Epoch progress: 2.91/8 (36.3%) |
| 2025-11-22 22:12:42,132 - __main__ - INFO - 📈 Epoch progress: 2.92/8 (36.5%) |
| 2025-11-22 22:13:05,631 - __main__ - INFO - 📈 Epoch progress: 2.93/8 (36.6%) |
| 2025-11-22 22:13:28,949 - __main__ - INFO - 📈 Epoch progress: 2.94/8 (36.7%) |
| 2025-11-22 22:13:52,766 - __main__ - INFO - 📈 Epoch progress: 2.95/8 (36.8%) |
| 2025-11-22 22:14:16,537 - __main__ - INFO - 📈 Epoch progress: 2.95/8 (36.9%) |
| 2025-11-22 22:14:40,633 - __main__ - INFO - 📈 Epoch progress: 2.96/8 (37.0%) |
| 2025-11-22 22:15:04,766 - __main__ - INFO - 📈 Epoch progress: 2.97/8 (37.2%) |
| 2025-11-22 22:15:28,401 - __main__ - INFO - 📈 Epoch progress: 2.98/8 (37.3%) |
| 2025-11-22 22:15:51,913 - __main__ - INFO - 📈 Epoch progress: 2.99/8 (37.4%) |
| 2025-11-22 22:16:16,816 - __main__ - INFO - 📈 Epoch progress: 3.00/8 (37.5%) |
| 2025-11-22 22:16:41,053 - __main__ - INFO - 📈 Epoch progress: 3.01/8 (37.6%) |
| 2025-11-22 22:17:05,221 - __main__ - INFO - 📈 Epoch progress: 3.02/8 (37.8%) |
| 2025-11-22 22:17:20,019 - __main__ - INFO - 📊 Evaluation 17: BLEU = 0.1803 |
| 2025-11-22 22:17:20,363 - __main__ - INFO - 📊 Evaluation at step 32000 |
| 2025-11-22 22:17:24,622 - __main__ - INFO - |
| 📤 Pushing checkpoint 32000 to HuggingFace Hub... |
| 2025-11-22 22:17:24,623 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 22:17:36,404 - __main__ - INFO - ✅ Checkpoint 32000 pushed successfully! |
| 2025-11-22 22:17:36,419 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-32000 |
| 2025-11-22 22:18:00,147 - __main__ - INFO - 📈 Epoch progress: 3.03/8 (37.9%) |
| 2025-11-22 22:18:24,208 - __main__ - INFO - 📈 Epoch progress: 3.04/8 (38.0%) |
| 2025-11-22 22:18:47,490 - __main__ - INFO - 📈 Epoch progress: 3.05/8 (38.1%) |
| 2025-11-22 22:19:11,236 - __main__ - INFO - 📈 Epoch progress: 3.06/8 (38.2%) |
| 2025-11-22 22:19:34,490 - __main__ - INFO - 📈 Epoch progress: 3.07/8 (38.3%) |
| 2025-11-22 22:19:58,382 - __main__ - INFO - 📈 Epoch progress: 3.08/8 (38.5%) |
| 2025-11-22 22:20:22,181 - __main__ - INFO - 📈 Epoch progress: 3.09/8 (38.6%) |
| 2025-11-22 22:20:46,069 - __main__ - INFO - 📈 Epoch progress: 3.10/8 (38.7%) |
| 2025-11-22 22:21:10,006 - __main__ - INFO - 📈 Epoch progress: 3.11/8 (38.8%) |
| 2025-11-22 22:21:33,569 - __main__ - INFO - 📈 Epoch progress: 3.11/8 (38.9%) |
| 2025-11-22 22:21:56,826 - __main__ - INFO - 📈 Epoch progress: 3.12/8 (39.1%) |
| 2025-11-22 22:22:20,689 - __main__ - INFO - 📈 Epoch progress: 3.13/8 (39.2%) |
| 2025-11-22 22:22:44,409 - __main__ - INFO - 📈 Epoch progress: 3.14/8 (39.3%) |
| 2025-11-22 22:23:08,312 - __main__ - INFO - 📈 Epoch progress: 3.15/8 (39.4%) |
| 2025-11-22 22:23:32,702 - __main__ - INFO - 📈 Epoch progress: 3.16/8 (39.5%) |
| 2025-11-22 22:23:55,904 - __main__ - INFO - 📈 Epoch progress: 3.17/8 (39.6%) |
| 2025-11-22 22:24:20,172 - __main__ - INFO - 📈 Epoch progress: 3.18/8 (39.8%) |
| 2025-11-22 22:24:44,449 - __main__ - INFO - 📈 Epoch progress: 3.19/8 (39.9%) |
| 2025-11-22 22:25:08,639 - __main__ - INFO - 📈 Epoch progress: 3.20/8 (40.0%) |
| 2025-11-22 22:25:32,297 - __main__ - INFO - 📈 Epoch progress: 3.21/8 (40.1%) |
| 2025-11-22 22:25:48,079 - __main__ - INFO - 📊 Evaluation 18: BLEU = 0.1742 |
| 2025-11-22 22:25:48,433 - __main__ - INFO - 📊 Evaluation at step 34000 |
| 2025-11-22 22:25:52,773 - __main__ - INFO - |
| 📤 Pushing checkpoint 34000 to HuggingFace Hub... |
| 2025-11-22 22:25:52,774 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 22:26:02,999 - __main__ - INFO - ✅ Checkpoint 34000 pushed successfully! |
| 2025-11-22 22:26:03,015 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-34000 |
| 2025-11-22 22:26:32,695 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:26:32,696 - __main__ - INFO - 📈 Epoch progress: 3.22/8 (40.2%) |
| 2025-11-22 22:26:33,015 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:26:57,007 - __main__ - INFO - 📈 Epoch progress: 3.23/8 (40.4%) |
| 2025-11-22 22:27:21,160 - __main__ - INFO - 📈 Epoch progress: 3.24/8 (40.5%) |
| 2025-11-22 22:27:45,784 - __main__ - INFO - 📈 Epoch progress: 3.25/8 (40.6%) |
| 2025-11-22 22:28:09,851 - __main__ - INFO - 📈 Epoch progress: 3.26/8 (40.7%) |
| 2025-11-22 22:28:33,187 - __main__ - INFO - 📈 Epoch progress: 3.27/8 (40.8%) |
| 2025-11-22 22:28:56,968 - __main__ - INFO - 📈 Epoch progress: 3.28/8 (40.9%) |
| 2025-11-22 22:29:20,427 - __main__ - INFO - 📈 Epoch progress: 3.28/8 (41.1%) |
| 2025-11-22 22:29:44,210 - __main__ - INFO - 📈 Epoch progress: 3.29/8 (41.2%) |
| 2025-11-22 22:30:08,158 - __main__ - INFO - 📈 Epoch progress: 3.30/8 (41.3%) |
| 2025-11-22 22:30:31,651 - __main__ - INFO - 📈 Epoch progress: 3.31/8 (41.4%) |
| 2025-11-22 22:30:55,813 - __main__ - INFO - 📈 Epoch progress: 3.32/8 (41.5%) |
| 2025-11-22 22:31:19,823 - __main__ - INFO - 📈 Epoch progress: 3.33/8 (41.7%) |
| 2025-11-22 22:31:43,557 - __main__ - INFO - 📈 Epoch progress: 3.34/8 (41.8%) |
| 2025-11-22 22:32:07,188 - __main__ - INFO - 📈 Epoch progress: 3.35/8 (41.9%) |
| 2025-11-22 22:32:37,715 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:32:37,718 - __main__ - INFO - 📈 Epoch progress: 3.36/8 (42.0%) |
| 2025-11-22 22:32:38,127 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:33:01,680 - __main__ - INFO - 📈 Epoch progress: 3.37/8 (42.1%) |
| 2025-11-22 22:33:25,806 - __main__ - INFO - 📈 Epoch progress: 3.38/8 (42.2%) |
| 2025-11-22 22:33:49,888 - __main__ - INFO - 📈 Epoch progress: 3.39/8 (42.4%) |
| 2025-11-22 22:34:13,802 - __main__ - INFO - 📈 Epoch progress: 3.40/8 (42.5%) |
| 2025-11-22 22:34:29,581 - __main__ - INFO - 📊 Evaluation 19: BLEU = 0.1791 |
| 2025-11-22 22:34:29,919 - __main__ - INFO - 📊 Evaluation at step 36000 |
| 2025-11-22 22:34:35,623 - __main__ - INFO - |
| 📤 Pushing checkpoint 36000 to HuggingFace Hub... |
| 2025-11-22 22:34:35,624 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 22:34:46,345 - __main__ - INFO - ✅ Checkpoint 36000 pushed successfully! |
| 2025-11-22 22:34:46,347 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-36000 |
| 2025-11-22 22:35:09,516 - __main__ - INFO - 📈 Epoch progress: 3.41/8 (42.6%) |
| 2025-11-22 22:35:33,257 - __main__ - INFO - 📈 Epoch progress: 3.42/8 (42.7%) |
| 2025-11-22 22:35:57,242 - __main__ - INFO - 📈 Epoch progress: 3.43/8 (42.8%) |
| 2025-11-22 22:36:21,693 - __main__ - INFO - 📈 Epoch progress: 3.44/8 (42.9%) |
| 2025-11-22 22:36:45,861 - __main__ - INFO - 📈 Epoch progress: 3.45/8 (43.1%) |
| 2025-11-22 22:37:10,943 - __main__ - INFO - 📈 Epoch progress: 3.45/8 (43.2%) |
| 2025-11-22 22:37:34,716 - __main__ - INFO - 📈 Epoch progress: 3.46/8 (43.3%) |
| 2025-11-22 22:37:59,177 - __main__ - INFO - 📈 Epoch progress: 3.47/8 (43.4%) |
| 2025-11-22 22:38:23,278 - __main__ - INFO - 📈 Epoch progress: 3.48/8 (43.5%) |
| 2025-11-22 22:38:47,089 - __main__ - INFO - 📈 Epoch progress: 3.49/8 (43.7%) |
| 2025-11-22 22:39:11,123 - __main__ - INFO - 📈 Epoch progress: 3.50/8 (43.8%) |
| 2025-11-22 22:39:35,715 - __main__ - INFO - 📈 Epoch progress: 3.51/8 (43.9%) |
| 2025-11-22 22:39:59,298 - __main__ - INFO - 📈 Epoch progress: 3.52/8 (44.0%) |
| 2025-11-22 22:40:22,780 - __main__ - INFO - 📈 Epoch progress: 3.53/8 (44.1%) |
| 2025-11-22 22:40:46,271 - __main__ - INFO - 📈 Epoch progress: 3.54/8 (44.2%) |
| 2025-11-22 22:41:10,602 - __main__ - INFO - 📈 Epoch progress: 3.55/8 (44.4%) |
| 2025-11-22 22:41:35,020 - __main__ - INFO - 📈 Epoch progress: 3.56/8 (44.5%) |
| 2025-11-22 22:41:58,896 - __main__ - INFO - 📈 Epoch progress: 3.57/8 (44.6%) |
| 2025-11-22 22:42:22,712 - __main__ - INFO - 📈 Epoch progress: 3.58/8 (44.7%) |
| 2025-11-22 22:42:46,271 - __main__ - INFO - 📈 Epoch progress: 3.59/8 (44.8%) |
| 2025-11-22 22:43:08,440 - __main__ - INFO - 📊 Evaluation 20: BLEU = 0.1705 |
| 2025-11-22 22:43:08,780 - __main__ - INFO - 📊 Evaluation at step 38000 |
| 2025-11-22 22:43:13,372 - __main__ - INFO - |
| 📤 Pushing checkpoint 38000 to HuggingFace Hub... |
| 2025-11-22 22:43:13,373 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 22:43:24,549 - __main__ - INFO - ✅ Checkpoint 38000 pushed successfully! |
| 2025-11-22 22:43:24,551 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-38000 |
| 2025-11-22 22:43:47,855 - __main__ - INFO - 📈 Epoch progress: 3.60/8 (45.0%) |
| 2025-11-22 22:44:12,036 - __main__ - INFO - 📈 Epoch progress: 3.61/8 (45.1%) |
| 2025-11-22 22:44:35,675 - __main__ - INFO - 📈 Epoch progress: 3.62/8 (45.2%) |
| 2025-11-22 22:44:54,430 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:45:06,039 - __main__ - INFO - 📈 Epoch progress: 3.62/8 (45.3%) |
| 2025-11-22 22:45:30,388 - __main__ - INFO - 📈 Epoch progress: 3.63/8 (45.4%) |
| 2025-11-22 22:45:54,473 - __main__ - INFO - 📈 Epoch progress: 3.64/8 (45.5%) |
| 2025-11-22 22:46:18,311 - __main__ - INFO - 📈 Epoch progress: 3.65/8 (45.7%) |
| 2025-11-22 22:46:42,472 - __main__ - INFO - 📈 Epoch progress: 3.66/8 (45.8%) |
| 2025-11-22 22:47:07,409 - __main__ - INFO - 📈 Epoch progress: 3.67/8 (45.9%) |
| 2025-11-22 22:47:30,886 - __main__ - INFO - 📈 Epoch progress: 3.68/8 (46.0%) |
| 2025-11-22 22:47:54,827 - __main__ - INFO - 📈 Epoch progress: 3.69/8 (46.1%) |
| 2025-11-22 22:48:18,009 - __main__ - INFO - 📈 Epoch progress: 3.70/8 (46.3%) |
| 2025-11-22 22:48:41,563 - __main__ - INFO - 📈 Epoch progress: 3.71/8 (46.4%) |
| 2025-11-22 22:49:06,015 - __main__ - INFO - 📈 Epoch progress: 3.72/8 (46.5%) |
| 2025-11-22 22:49:30,312 - __main__ - INFO - 📈 Epoch progress: 3.73/8 (46.6%) |
| 2025-11-22 22:49:54,384 - __main__ - INFO - 📈 Epoch progress: 3.74/8 (46.7%) |
| 2025-11-22 22:50:18,637 - __main__ - INFO - 📈 Epoch progress: 3.75/8 (46.8%) |
| 2025-11-22 22:50:42,692 - __main__ - INFO - 📈 Epoch progress: 3.76/8 (47.0%) |
| 2025-11-22 22:51:06,477 - __main__ - INFO - 📈 Epoch progress: 3.77/8 (47.1%) |
| 2025-11-22 22:51:30,876 - __main__ - INFO - 📈 Epoch progress: 3.78/8 (47.2%) |
| 2025-11-22 22:51:49,849 - __main__ - INFO - 📊 Evaluation 21: BLEU = 0.1801 |
| 2025-11-22 22:51:50,268 - __main__ - INFO - 📊 Evaluation at step 40000 |
| 2025-11-22 22:51:55,274 - __main__ - INFO - |
| 📤 Pushing checkpoint 40000 to HuggingFace Hub... |
| 2025-11-22 22:51:55,275 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 22:52:05,418 - __main__ - INFO - ✅ Checkpoint 40000 pushed successfully! |
| 2025-11-22 22:52:05,419 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-40000 |
| 2025-11-22 22:52:28,376 - __main__ - INFO - 📈 Epoch progress: 3.79/8 (47.3%) |
| 2025-11-22 22:52:52,172 - __main__ - INFO - 📈 Epoch progress: 3.79/8 (47.4%) |
| 2025-11-22 22:53:16,023 - __main__ - INFO - 📈 Epoch progress: 3.80/8 (47.6%) |
| 2025-11-22 22:53:39,673 - __main__ - INFO - 📈 Epoch progress: 3.81/8 (47.7%) |
| 2025-11-22 22:54:03,075 - __main__ - INFO - 📈 Epoch progress: 3.82/8 (47.8%) |
| 2025-11-22 22:54:27,337 - __main__ - INFO - 📈 Epoch progress: 3.83/8 (47.9%) |
| 2025-11-22 22:54:57,783 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:54:57,790 - __main__ - INFO - 📈 Epoch progress: 3.84/8 (48.0%) |
| 2025-11-22 22:54:58,152 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-22 22:55:21,366 - __main__ - INFO - 📈 Epoch progress: 3.85/8 (48.1%) |
| 2025-11-22 22:55:45,172 - __main__ - INFO - 📈 Epoch progress: 3.86/8 (48.3%) |
| 2025-11-22 22:56:08,569 - __main__ - INFO - 📈 Epoch progress: 3.87/8 (48.4%) |
| 2025-11-22 22:56:32,826 - __main__ - INFO - 📈 Epoch progress: 3.88/8 (48.5%) |
| 2025-11-22 22:56:56,673 - __main__ - INFO - 📈 Epoch progress: 3.89/8 (48.6%) |
| 2025-11-22 22:57:20,679 - __main__ - INFO - 📈 Epoch progress: 3.90/8 (48.7%) |
| 2025-11-22 22:57:44,496 - __main__ - INFO - 📈 Epoch progress: 3.91/8 (48.8%) |
| 2025-11-22 22:58:08,328 - __main__ - INFO - 📈 Epoch progress: 3.92/8 (49.0%) |
| 2025-11-22 22:58:31,781 - __main__ - INFO - 📈 Epoch progress: 3.93/8 (49.1%) |
| 2025-11-22 22:58:55,466 - __main__ - INFO - 📈 Epoch progress: 3.94/8 (49.2%) |
| 2025-11-22 22:59:19,046 - __main__ - INFO - 📈 Epoch progress: 3.95/8 (49.3%) |
| 2025-11-22 22:59:42,251 - __main__ - INFO - 📈 Epoch progress: 3.96/8 (49.4%) |
| 2025-11-22 23:00:06,289 - __main__ - INFO - 📈 Epoch progress: 3.96/8 (49.6%) |
| 2025-11-22 23:00:33,348 - __main__ - INFO - 📊 Evaluation 22: BLEU = 0.1746 |
| 2025-11-22 23:00:34,193 - __main__ - INFO - 📊 Evaluation at step 42000 |
| 2025-11-22 23:00:38,079 - __main__ - INFO - |
| 📤 Pushing checkpoint 42000 to HuggingFace Hub... |
| 2025-11-22 23:00:38,080 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 23:00:48,483 - __main__ - INFO - ✅ Checkpoint 42000 pushed successfully! |
| 2025-11-22 23:00:48,485 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/checkpoints/checkpoint-42000 |
| 2025-11-22 23:00:51,556 - __main__ - INFO - ✅ MLflow: Training finished - status_code=2.0 (FINISHED) |
| 2025-11-22 23:00:51,853 - __main__ - INFO - |
| 💾 Saving model to models/finetuned-sw-en-phase1 |
| 2025-11-22 23:00:53,397 - __main__ - INFO - ✅ Training complete! |
| 2025-11-22 23:00:53,398 - __main__ - INFO - |
| 📊 STEP 3: Final Evaluation |
| 2025-11-22 23:00:53,761 - __main__ - INFO - ✅ MLflow: Final evaluation starting - status_code=3.0 |
| 2025-11-22 23:00:53,762 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 23:00:53,762 - __main__ - INFO - 📊 Running FINAL evaluation on test set... |
| 2025-11-22 23:00:53,763 - __main__ - INFO - ====================================================================== |
| 2025-11-22 23:00:53,764 - __main__ - INFO - Tokenizing test dataset... |
| 2025-11-22 23:00:55,719 - datasets.fingerprint - WARNING - Parameter 'function'=<bound method TranslationTrainer.preprocess_function of <__main__.TranslationTrainer object at 0x7e5c556f96d0>> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown. |
| 2025-11-22 23:49:59,784 - __main__ - INFO - 📊 Evaluation 23: BLEU = 0.1867 |
| 2025-11-22 23:50:00,135 - __main__ - INFO - 📊 Evaluation at step 42000 |
| 2025-11-22 23:50:00,257 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-22 23:50:00,313 - __main__ - INFO - Computing predictions for COMET (1000 samples, batch_size=32)... |
| 2025-11-22 23:50:00,319 - __main__ - INFO - Batch 1/32... |
| 2025-11-22 23:50:04,256 - __main__ - INFO - Batch 5/32... |
| 2025-11-22 23:50:05,629 - __main__ - INFO - Batch 10/32... |
| 2025-11-22 23:50:06,935 - __main__ - INFO - Batch 15/32... |
| 2025-11-22 23:50:08,526 - __main__ - INFO - Batch 20/32... |
| 2025-11-22 23:50:12,080 - __main__ - INFO - Batch 25/32... |
| 2025-11-22 23:50:13,411 - __main__ - INFO - Batch 30/32... |
| 2025-11-22 23:50:13,939 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-22 23:50:13,949 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-22 23:50:13,961 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-22 23:50:13,962 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-22 23:50:13,964 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-22 23:50:19,326 - __main__ - INFO - ✅ COMET-QE: 0.2124 ± 0.2721 |
| 2025-11-22 23:50:19,327 - __main__ - INFO - |
| ✅ FINAL Test Results: |
| 2025-11-22 23:50:19,328 - __main__ - INFO - BLEU: 0.1867 |
| 2025-11-22 23:50:19,328 - __main__ - INFO - chrF: 38.35 |
| 2025-11-22 23:50:19,329 - __main__ - INFO - COMET-QE: 0.2124 |
| 2025-11-22 23:50:19,329 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 23:50:19,330 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 23:50:19,331 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-22 23:50:19,331 - __main__ - INFO - ====================================================================== |
| 2025-11-22 23:50:19,331 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-22 23:50:19,332 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-22 23:50:20,433 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-22 23:50:20,518 - __main__ - INFO - ✅ Domain chrF: 1.22 |
| 2025-11-22 23:50:20,518 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-22 23:50:20,526 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-22 23:50:20,535 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-22 23:50:20,536 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-22 23:50:20,537 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-22 23:50:23,127 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 |
| 2025-11-22 23:50:23,127 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-22 23:50:23,128 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-22 23:50:23,129 - __main__ - INFO - chrF: 1.22 |
| 2025-11-22 23:50:23,129 - __main__ - INFO - COMET-QE: 0.0001 |
| 2025-11-22 23:50:23,129 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 23:50:23,137 - __main__ - INFO - 💾 Final results saved to models/finetuned-sw-en-phase1/test_results.json |
| 2025-11-22 23:50:23,141 - __main__ - INFO - ✅ Verified test_results.json with 14 metrics |
| 2025-11-22 23:50:23,141 - __main__ - INFO - |
| 📊 Logging all metrics to TensorBoard... |
| 2025-11-22 23:50:23,278 - __main__ - INFO - ✅ All metrics logged to TensorBoard |
| 2025-11-22 23:50:23,278 - __main__ - INFO - Baseline metrics: 13 |
| 2025-11-22 23:50:23,279 - __main__ - INFO - Final test metrics: 10 |
| 2025-11-22 23:50:23,280 - __main__ - INFO - Final domain metrics: 4 |
| 2025-11-22 23:50:23,281 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 23:50:23,281 - __main__ - INFO - 📊 BASELINE vs FINE-TUNED Comparison |
| 2025-11-22 23:50:23,282 - __main__ - INFO - ====================================================================== |
| 2025-11-22 23:50:23,282 - __main__ - INFO - |
| 📈 Test Set Metrics: |
| 2025-11-22 23:50:23,283 - __main__ - INFO - Metric Baseline Fine-tuned Improvement |
| 2025-11-22 23:50:23,283 - __main__ - INFO - ------------------------------------------------------------ |
| 2025-11-22 23:50:23,284 - __main__ - INFO - eval_bleu 0.0000 0.1867 N/A |
| 2025-11-22 23:50:23,290 - __main__ - INFO - eval_chrf 0.0000 38.3529 N/A |
| 2025-11-22 23:50:23,291 - __main__ - INFO - eval_comet 0.0000 0.2124 N/A |
| 2025-11-22 23:50:23,291 - __main__ - INFO - |
| 📈 Domain Metrics (Real Call Transcriptions): |
| 2025-11-22 23:50:23,292 - __main__ - INFO - Metric Baseline Fine-tuned Improvement |
| 2025-11-22 23:50:23,293 - __main__ - INFO - ------------------------------------------------------------ |
| 2025-11-22 23:50:23,293 - __main__ - INFO - domain_bleu 0.0000 0.0000 N/A |
| 2025-11-22 23:50:23,294 - __main__ - INFO - domain_chrf 0.9342 1.2250 +31.1% |
| 2025-11-22 23:50:23,294 - __main__ - INFO - domain_comet 0.0000 0.0001 +33.3% |
| 2025-11-22 23:50:23,295 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 23:50:23,664 - __main__ - INFO - ✅ Logged 14 final metrics to MLflow |
| 2025-11-22 23:50:23,665 - __main__ - INFO - Visit MLflow: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 23:50:23,666 - __main__ - INFO - 📊 Captured 863 training log entries |
| 2025-11-22 23:50:23,718 - __main__ - INFO - 💾 Saved comprehensive metrics to models/finetuned-sw-en-phase1/training_metrics.json |
| 2025-11-22 23:50:23,734 - __main__ - INFO - 💾 Saved training config to models/finetuned-sw-en-phase1/training_config_used.json |
| 2025-11-22 23:50:23,758 - __main__ - INFO - 💾 Copied training.log to output directory |
| 2025-11-22 23:50:23,759 - __main__ - INFO - |
| 🤖 HUGGINGFACE HUB |
| 2025-11-22 23:50:23,760 - __main__ - INFO - ℹ️ Checkpoints were pushed during training (push_strategy='checkpoint') |
| 2025-11-22 23:50:23,761 - __main__ - INFO - 📤 Pushing final model to hub... |
| 2025-11-22 23:50:23,761 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 23:50:23,762 - __main__ - INFO - 📤 Uploading model to HuggingFace Hub... |
| 2025-11-22 23:50:23,763 - __main__ - INFO - ====================================================================== |
| 2025-11-22 23:50:23,763 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 23:50:23,774 - __main__ - INFO - ✅ Model card generated |
| 2025-11-22 23:50:24,095 - huggingface_hub._login - WARNING - Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you've just configured. |
| 2025-11-22 23:50:24,096 - __main__ - INFO - Uploading files... |
| 2025-11-22 23:50:24,170 - __main__ - INFO - ✅ Repository created/verified: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 23:50:24,174 - __main__ - INFO - 📊 Including TensorBoard logs (5 files) |
| 2025-11-22 23:50:24,176 - __main__ - INFO - 📁 Uploading 21 items from models/finetuned-sw-en-phase1 |
| 2025-11-22 23:50:24,177 - __main__ - INFO - 📄 README.md |
| 2025-11-22 23:50:24,178 - __main__ - INFO - 📄 training.log |
| 2025-11-22 23:50:24,179 - __main__ - INFO - 📄 training_config_used.json |
| 2025-11-22 23:50:24,181 - __main__ - INFO - 📄 training_metrics.json |
| 2025-11-22 23:50:24,182 - __main__ - INFO - 📄 test_results.json |
| 2025-11-22 23:50:24,183 - __main__ - INFO - 📄 train_results.json |
| 2025-11-22 23:50:24,184 - __main__ - INFO - 📄 translation_config.json |
| 2025-11-22 23:50:24,185 - __main__ - INFO - 📄 training_args.bin |
| 2025-11-22 23:50:24,186 - __main__ - INFO - 📄 target.spm |
| 2025-11-22 23:50:24,187 - __main__ - INFO - 📄 source.spm |
| 2025-11-22 23:50:24,188 - __main__ - INFO - 📄 vocab.json |
| 2025-11-22 23:50:24,189 - __main__ - INFO - 📄 special_tokens_map.json |
| 2025-11-22 23:50:24,191 - __main__ - INFO - 📄 tokenizer_config.json |
| 2025-11-22 23:50:24,192 - __main__ - INFO - 📄 model.safetensors |
| 2025-11-22 23:50:24,193 - __main__ - INFO - 📄 generation_config.json |
| 2025-11-22 23:50:24,194 - __main__ - INFO - 📄 config.json |
| 2025-11-22 23:50:24,195 - __main__ - INFO - 📂 checkpoint-42000 |
| 2025-11-22 23:50:24,196 - __main__ - INFO - 📂 checkpoint-40000 |
| 2025-11-22 23:50:24,197 - __main__ - INFO - 📂 checkpoint-32000 |
| 2025-11-22 23:50:24,198 - __main__ - INFO - 📄 baseline_results.json |
| 2025-11-22 23:50:24,198 - __main__ - INFO - 📂 logs |
| 2025-11-22 23:50:43,662 - __main__ - INFO - |
| ✅ Model uploaded successfully! |
| 2025-11-22 23:50:43,664 - __main__ - INFO - 🔗 View model at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-22 23:50:43,667 - __main__ - INFO - 📊 TensorBoard logs at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase1/tree/main/logs |
| 2025-11-22 23:50:43,667 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-22 23:50:43,668 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 23:50:43,669 - __main__ - INFO - ✅ Pipeline completed successfully! |
| 2025-11-22 23:50:43,669 - __main__ - INFO - ====================================================================== |
| 2025-11-22 23:50:43,670 - __main__ - INFO - |
| 📋 PIPELINE SUMMARY |
| 2025-11-22 23:50:43,671 - __main__ - INFO - ====================================================================== |
| 2025-11-22 23:50:43,671 - __main__ - INFO - |
| ✓ Baseline Evaluation |
| 2025-11-22 23:50:43,672 - __main__ - INFO - - Test set: 159518 samples |
| 2025-11-22 23:50:43,672 - __main__ - INFO - - Domain data: 5 samples |
| 2025-11-22 23:50:43,673 - __main__ - INFO - |
| ✓ Training |
| 2025-11-22 23:50:43,674 - __main__ - INFO - - Epochs: 8 |
| 2025-11-22 23:50:43,674 - __main__ - INFO - - Train samples: 2,711,789 |
| 2025-11-22 23:50:43,675 - __main__ - INFO - - Validation samples: 319,034 |
| 2025-11-22 23:50:43,676 - __main__ - INFO - |
| ✓ Final Evaluation |
| 2025-11-22 23:50:43,676 - __main__ - INFO - - Test set BLEU: 0.1867 |
| 2025-11-22 23:50:43,677 - __main__ - INFO - - Domain BLEU: 0.0000 |
| 2025-11-22 23:50:43,677 - __main__ - INFO - |
| 📁 Output Files |
| 2025-11-22 23:50:43,679 - __main__ - INFO - - Models: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1 |
| 2025-11-22 23:50:43,679 - __main__ - INFO - - Baseline results: models/finetuned-sw-en-phase1/baseline_results.json |
| 2025-11-22 23:50:43,680 - __main__ - INFO - - Test results: models/finetuned-sw-en-phase1/test_results.json |
| 2025-11-22 23:50:43,681 - __main__ - INFO - - Training logs: /workspace/franklin/openchs_rnd/tasks/translation/swahili/training.log |
| 2025-11-22 23:50:43,681 - __main__ - INFO - |
| 📈 MLflow Tracking |
| 2025-11-22 23:50:43,682 - __main__ - INFO - - URL: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 23:50:43,683 - __main__ - INFO - - Run ID: 8b5deaed89ec48e986246e5b801158e3 |
| 2025-11-22 23:50:43,683 - __main__ - INFO - |
| 📊 TensorBoard Logs |
| 2025-11-22 23:50:43,684 - __main__ - INFO - - View with: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase1/logs |
| 2025-11-22 23:50:43,684 - __main__ - INFO - - All metrics logged: baseline, test, domain, and COMET scores |
| 2025-11-22 23:50:43,686 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-22 23:50:43,686 - __main__ - INFO - 📈 MLflow Tracking: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-22 23:50:43,687 - __main__ - INFO - |
| Status codes: |
| 2025-11-22 23:50:43,687 - __main__ - INFO - - 0.5 = Baseline evaluation |
| 2025-11-22 23:50:43,688 - __main__ - INFO - - 1.0 = Training started |
| 2025-11-22 23:50:43,689 - __main__ - INFO - - 2.0 = Training finished |
| 2025-11-22 23:50:43,689 - __main__ - INFO - - 3.0 = Final evaluation |
| 2025-11-22 23:50:44,048 - __main__ - INFO - |
| ✅ MLflow: Pipeline completed - status_code=4.0 |
| 2025-11-22 23:50:44,889 - __main__ - INFO - ✅ MLflow run ended: 8b5deaed89ec48e986246e5b801158e3 |
| 2025-11-22 23:50:44,891 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 04:05:43,732 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-23 04:05:43,743 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-23 04:05:43,744 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-23 04:05:43,745 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-23 04:05:43,747 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-23 04:05:43,748 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-23 04:05:43,748 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 04:05:43,749 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-23 04:05:43,749 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-23 04:05:43,749 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-23 04:05:43,750 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-23 04:05:43,751 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-23 04:05:43,760 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-23 04:05:43,761 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 04:05:44,443 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-23 04:05:45,579 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251123_040544 |
| 2025-11-23 04:05:45,580 - __main__ - INFO - Run ID: e70d46acae184a60a902a20b85021668 |
| 2025-11-23 04:05:45,581 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 04:05:45,962 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow |
| 2025-11-23 04:05:45,965 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-23 04:05:45,976 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-23 04:05:45,977 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-23 04:05:45,978 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-23 04:05:54,359 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-23 04:05:54,361 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-23 04:05:54,696 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-23 04:05:54,923 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-23 04:05:54,924 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-23 04:05:57,916 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-23 04:06:00,884 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:03,675 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:06,082 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:08,442 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:10,807 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:13,193 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:15,538 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:17,888 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:20,255 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:22,631 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:25,011 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:27,409 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:29,801 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:32,196 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:34,590 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:36,955 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:39,341 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:41,730 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:44,125 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:46,526 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:48,904 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:51,289 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:53,684 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:56,086 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:58,486 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-23 04:06:58,719 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-23 04:06:58,754 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-23 04:07:00,804 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-23 04:07:03,765 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:06,444 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:08,822 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:11,186 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:13,575 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:15,965 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:18,459 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:20,871 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:23,259 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:25,654 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:28,055 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:30,458 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:32,855 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:35,252 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:37,634 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:40,067 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 |
| 2025-11-23 04:07:42,278 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) |
| 2025-11-23 04:07:42,325 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 |
| 2025-11-23 04:07:42,326 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-23 04:07:42,327 - __main__ - INFO - Sample 1: |
| 2025-11-23 04:07:42,327 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... |
| 2025-11-23 04:07:42,328 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... |
| 2025-11-23 04:07:42,328 - __main__ - INFO - Sample 2: |
| 2025-11-23 04:07:42,329 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... |
| 2025-11-23 04:07:42,329 - __main__ - INFO - Target (6 words): Job had said no such thing.... |
| 2025-11-23 04:07:42,330 - __main__ - INFO - Sample 3: |
| 2025-11-23 04:07:42,331 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... |
| 2025-11-23 04:07:42,331 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... |
| 2025-11-23 04:07:47,761 - __main__ - INFO - Filtering statistics: {'kept': 4202243} |
| 2025-11-23 04:07:47,762 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 |
| 2025-11-23 04:07:52,179 - __main__ - INFO - Removed 518 duplicates |
| 2025-11-23 04:07:52,414 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 |
| 2025-11-23 04:07:59,017 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 04:07:59,023 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-23 04:07:59,025 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-23 04:07:59,027 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-23 04:07:59,027 - __main__ - INFO - Test: 210,087 samples |
| 2025-11-23 04:07:59,028 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 04:07:59,463 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-23 04:11:21,132 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-23 04:11:21,139 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-23 04:11:21,140 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-23 04:11:21,141 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-23 04:11:21,143 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-23 04:11:21,144 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-23 04:11:21,144 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 04:11:21,145 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-23 04:11:21,145 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-23 04:11:21,146 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-23 04:11:21,146 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-23 04:11:21,147 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-23 04:11:21,158 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-23 04:11:21,159 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 04:11:21,913 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-23 04:11:22,843 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251123_041122 |
| 2025-11-23 04:11:22,845 - __main__ - INFO - Run ID: e2e357df5cbe463abba0139445600679 |
| 2025-11-23 04:11:22,845 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 04:11:23,226 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow |
| 2025-11-23 04:11:23,231 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-23 04:11:23,239 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-23 04:11:23,241 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-23 04:11:23,241 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-23 04:11:26,716 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-23 04:11:26,717 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-23 04:11:27,122 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-23 04:11:27,390 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-23 04:11:27,393 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-23 04:11:28,190 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-23 04:11:31,094 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:33,656 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:36,508 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:38,901 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:41,318 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:43,721 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:46,121 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:48,539 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:50,925 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:53,348 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:55,767 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-23 04:11:58,204 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:00,636 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:03,051 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:05,459 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:07,868 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:10,285 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:12,718 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:15,153 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:17,596 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:20,000 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:22,421 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:24,843 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:27,269 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:29,700 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-23 04:12:29,952 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-23 04:12:29,998 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-23 04:12:30,726 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-23 04:12:33,185 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:35,566 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:37,932 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:40,292 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:42,663 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:45,025 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:47,392 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:49,787 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:52,193 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:54,609 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:57,036 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 |
| 2025-11-23 04:12:59,440 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 |
| 2025-11-23 04:13:01,849 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 |
| 2025-11-23 04:13:04,266 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 |
| 2025-11-23 04:13:06,685 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 |
| 2025-11-23 04:13:09,101 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 |
| 2025-11-23 04:13:11,345 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) |
| 2025-11-23 04:13:11,386 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 |
| 2025-11-23 04:13:11,387 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-23 04:13:11,387 - __main__ - INFO - Sample 1: |
| 2025-11-23 04:13:11,388 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... |
| 2025-11-23 04:13:11,388 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... |
| 2025-11-23 04:13:11,389 - __main__ - INFO - Sample 2: |
| 2025-11-23 04:13:11,389 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... |
| 2025-11-23 04:13:11,389 - __main__ - INFO - Target (6 words): Job had said no such thing.... |
| 2025-11-23 04:13:11,390 - __main__ - INFO - Sample 3: |
| 2025-11-23 04:13:11,391 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... |
| 2025-11-23 04:13:11,391 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... |
| 2025-11-23 04:13:16,729 - __main__ - INFO - Filtering statistics: {'kept': 4202243} |
| 2025-11-23 04:13:16,730 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 |
| 2025-11-23 04:13:21,152 - __main__ - INFO - Removed 518 duplicates |
| 2025-11-23 04:13:21,390 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 |
| 2025-11-23 04:13:27,939 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 04:13:27,941 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-23 04:13:27,942 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-23 04:13:27,942 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-23 04:13:27,943 - __main__ - INFO - Test: 210,087 samples |
| 2025-11-23 04:13:27,943 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 04:13:28,390 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-23 04:16:33,982 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-23 04:16:33,984 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-23 04:16:33,985 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-23 04:16:33,986 - __main__ - INFO - Test: 210,087 samples |
|
|
| 2025-11-23 04:16:35,491 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) |
| 2025-11-23 04:16:35,492 - __main__ - INFO - Skipping tokenization - this saves significant time! |
| 2025-11-23 04:16:35,493 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... |
| 2025-11-23 04:17:11,756 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-23 04:17:11,757 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-23 04:17:11,859 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-23 04:17:12,851 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-23 04:17:18,905 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-23 04:17:19,462 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-23 04:17:19,463 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-23 04:17:19,494 - __main__ - INFO - 📊 Limiting evaluation dataset from 420225 to 2000 samples |
| 2025-11-23 04:17:19,516 - __main__ - INFO - ✅ Trainer initialized |
| 2025-11-23 04:17:19,517 - __main__ - INFO - |
| 🔬 STEP 1: Baseline Evaluation |
| 2025-11-23 04:17:20,164 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 |
| 2025-11-23 04:17:20,165 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 04:17:20,165 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... |
| 2025-11-23 04:17:20,166 - __main__ - INFO - ====================================================================== |
| 2025-11-23 04:17:52,543 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1215 |
| 2025-11-23 04:17:52,544 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-23 04:17:52,576 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... |
| 2025-11-23 04:17:52,578 - __main__ - INFO - Batch 1/16... |
| 2025-11-23 04:17:53,920 - __main__ - INFO - Batch 5/16... |
| 2025-11-23 04:17:54,977 - __main__ - INFO - Batch 10/16... |
| 2025-11-23 04:17:56,189 - __main__ - INFO - Batch 15/16... |
| 2025-11-23 04:17:56,576 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-23 04:17:56,588 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-23 04:17:56,596 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-23 04:17:56,597 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-23 04:17:56,598 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision |
| 2025-11-23 04:17:56,599 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-23 04:17:59,616 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 |
| 2025-11-23 04:17:59,619 - __main__ - INFO - |
| ✅ Baseline Results (Test Set): |
| 2025-11-23 04:17:59,620 - __main__ - INFO - BLEU: 0.1215 |
| 2025-11-23 04:17:59,622 - __main__ - INFO - chrF: 32.36 |
| 2025-11-23 04:17:59,622 - __main__ - INFO - COMET: 0.1977 |
| 2025-11-23 04:17:59,622 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 04:17:59,623 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 04:17:59,624 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-23 04:17:59,624 - __main__ - INFO - ====================================================================== |
| 2025-11-23 04:17:59,625 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-23 04:17:59,626 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-23 04:18:00,687 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-23 04:18:00,771 - __main__ - INFO - ✅ Domain chrF: 1.22 |
| 2025-11-23 04:18:00,771 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-23 04:18:00,781 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-23 04:18:00,788 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-23 04:18:00,789 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-23 04:18:00,791 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-23 04:18:02,815 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 |
| 2025-11-23 04:18:02,818 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-23 04:18:02,820 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-23 04:18:02,821 - __main__ - INFO - chrF: 1.22 |
| 2025-11-23 04:18:02,821 - __main__ - INFO - COMET-QE: 0.0001 |
| 2025-11-23 04:18:02,822 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 04:18:03,214 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 |
| 2025-11-23 04:18:03,234 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json |
| 2025-11-23 04:18:03,239 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics |
| 2025-11-23 04:18:03,240 - __main__ - INFO - |
| 🚀 STEP 2: Fine-tuning |
| 2025-11-23 04:18:03,240 - __main__ - INFO - Metrics are being logged to MLflow in real-time |
| 2025-11-23 04:18:03,241 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 04:18:03,241 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs |
| 2025-11-23 04:18:03,242 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') |
| 2025-11-23 04:18:03,243 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 04:18:03,243 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 04:18:03,244 - __main__ - INFO - 🚀 Starting training... |
| 2025-11-23 04:18:03,244 - __main__ - INFO - ====================================================================== |
| 2025-11-23 04:18:04,245 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) |
| 2025-11-23 04:18:25,258 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.1%) |
| 2025-11-23 04:18:45,682 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.2%) |
| 2025-11-23 04:19:06,543 - __main__ - INFO - 📈 Epoch progress: 0.02/6 (0.4%) |
| 2025-11-23 04:19:26,980 - __main__ - INFO - 📈 Epoch progress: 0.03/6 (0.5%) |
| 2025-11-23 04:19:47,541 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.6%) |
| 2025-11-23 04:20:07,833 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.7%) |
| 2025-11-23 04:20:29,177 - __main__ - INFO - 📈 Epoch progress: 0.05/6 (0.8%) |
| 2025-11-23 04:20:49,579 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.0%) |
| 2025-11-23 04:21:10,348 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.1%) |
| 2025-11-23 04:21:30,953 - __main__ - INFO - 📈 Epoch progress: 0.07/6 (1.2%) |
| 2025-11-23 04:21:51,628 - __main__ - INFO - 📈 Epoch progress: 0.08/6 (1.3%) |
| 2025-11-23 04:22:12,093 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.4%) |
| 2025-11-23 04:22:35,662 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 04:22:35,664 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.6%) |
| 2025-11-23 04:22:35,984 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 04:22:56,478 - __main__ - INFO - 📈 Epoch progress: 0.10/6 (1.7%) |
| 2025-11-23 04:23:17,467 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.8%) |
| 2025-11-23 04:23:38,091 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.9%) |
| 2025-11-23 04:23:58,922 - __main__ - INFO - 📈 Epoch progress: 0.12/6 (2.0%) |
| 2025-11-23 04:24:19,633 - __main__ - INFO - 📈 Epoch progress: 0.13/6 (2.2%) |
| 2025-11-23 04:24:40,816 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.3%) |
| 2025-11-23 04:25:01,487 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.4%) |
| 2025-11-23 04:25:50,552 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1169 |
| 2025-11-23 04:25:50,914 - __main__ - INFO - 📊 Evaluation at step 2000 |
| 2025-11-23 04:25:55,509 - __main__ - INFO - ✅ Repository brendaogutu/mul-sw-en-translation-phase2 ready |
| 2025-11-23 04:25:55,558 - __main__ - INFO - |
| 📤 Pushing checkpoint 2000 to HuggingFace Hub... |
| 2025-11-23 04:25:55,564 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 04:26:08,358 - __main__ - INFO - ✅ Checkpoint 2000 pushed successfully! |
| 2025-11-23 04:26:08,360 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-2000 |
| 2025-11-23 04:26:29,312 - __main__ - INFO - 📈 Epoch progress: 0.15/6 (2.5%) |
| 2025-11-23 04:26:49,842 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.6%) |
| 2025-11-23 04:27:10,452 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.7%) |
| 2025-11-23 04:27:30,891 - __main__ - INFO - 📈 Epoch progress: 0.17/6 (2.9%) |
| 2025-11-23 04:27:51,404 - __main__ - INFO - 📈 Epoch progress: 0.18/6 (3.0%) |
| 2025-11-23 04:28:11,815 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.1%) |
| 2025-11-23 04:28:31,947 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.2%) |
| 2025-11-23 04:28:52,529 - __main__ - INFO - 📈 Epoch progress: 0.20/6 (3.3%) |
| 2025-11-23 04:29:12,902 - __main__ - INFO - 📈 Epoch progress: 0.21/6 (3.5%) |
| 2025-11-23 04:29:34,164 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.6%) |
| 2025-11-23 04:29:54,671 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.7%) |
| 2025-11-23 04:30:15,398 - __main__ - INFO - 📈 Epoch progress: 0.23/6 (3.8%) |
| 2025-11-23 04:30:36,458 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (3.9%) |
| 2025-11-23 04:30:57,150 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (4.1%) |
| 2025-11-23 04:31:17,954 - __main__ - INFO - 📈 Epoch progress: 0.25/6 (4.2%) |
| 2025-11-23 04:31:38,767 - __main__ - INFO - 📈 Epoch progress: 0.26/6 (4.3%) |
| 2025-11-23 04:31:59,507 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.4%) |
| 2025-11-23 04:32:20,095 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.5%) |
| 2025-11-23 04:32:35,746 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 04:32:45,647 - __main__ - INFO - 📈 Epoch progress: 0.28/6 (4.7%) |
| 2025-11-23 04:33:06,504 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.8%) |
| 2025-11-23 04:33:52,383 - __main__ - INFO - 📊 Evaluation 3: BLEU = 0.1186 |
| 2025-11-23 04:33:52,744 - __main__ - INFO - 📊 Evaluation at step 4000 |
| 2025-11-23 04:33:56,093 - __main__ - INFO - |
| 📤 Pushing checkpoint 4000 to HuggingFace Hub... |
| 2025-11-23 04:33:56,095 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 04:34:06,396 - __main__ - INFO - ✅ Checkpoint 4000 pushed successfully! |
| 2025-11-23 04:34:06,398 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-4000 |
| 2025-11-23 04:34:26,610 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.9%) |
| 2025-11-23 04:34:47,206 - __main__ - INFO - 📈 Epoch progress: 0.30/6 (5.0%) |
| 2025-11-23 04:35:07,510 - __main__ - INFO - 📈 Epoch progress: 0.31/6 (5.1%) |
| 2025-11-23 04:35:27,928 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.3%) |
| 2025-11-23 04:35:48,227 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.4%) |
| 2025-11-23 04:36:08,893 - __main__ - INFO - 📈 Epoch progress: 0.33/6 (5.5%) |
| 2025-11-23 04:36:29,539 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.6%) |
| 2025-11-23 04:36:50,343 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.7%) |
| 2025-11-23 04:37:10,968 - __main__ - INFO - 📈 Epoch progress: 0.35/6 (5.9%) |
| 2025-11-23 04:37:31,505 - __main__ - INFO - 📈 Epoch progress: 0.36/6 (6.0%) |
| 2025-11-23 04:37:52,113 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.1%) |
| 2025-11-23 04:38:12,698 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.2%) |
| 2025-11-23 04:38:33,129 - __main__ - INFO - 📈 Epoch progress: 0.38/6 (6.3%) |
| 2025-11-23 04:38:54,508 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.5%) |
| 2025-11-23 04:39:15,066 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.6%) |
| 2025-11-23 04:39:35,569 - __main__ - INFO - 📈 Epoch progress: 0.40/6 (6.7%) |
| 2025-11-23 04:39:56,468 - __main__ - INFO - 📈 Epoch progress: 0.41/6 (6.8%) |
| 2025-11-23 04:40:16,968 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (6.9%) |
| 2025-11-23 04:40:29,982 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 04:40:40,394 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (7.0%) |
| 2025-11-23 04:41:00,393 - __main__ - INFO - 📈 Epoch progress: 0.43/6 (7.2%) |
| 2025-11-23 04:41:48,848 - __main__ - INFO - 📊 Evaluation 4: BLEU = 0.1098 |
| 2025-11-23 04:41:49,164 - __main__ - INFO - 📊 Evaluation at step 6000 |
| 2025-11-23 04:41:52,562 - __main__ - INFO - |
| 📤 Pushing checkpoint 6000 to HuggingFace Hub... |
| 2025-11-23 04:41:52,563 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 04:42:01,604 - __main__ - INFO - ✅ Checkpoint 6000 pushed successfully! |
| 2025-11-23 04:42:01,616 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-6000 |
| 2025-11-23 04:42:21,350 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.3%) |
| 2025-11-23 04:42:41,788 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.4%) |
| 2025-11-23 04:43:02,549 - __main__ - INFO - 📈 Epoch progress: 0.45/6 (7.5%) |
| 2025-11-23 04:43:23,339 - __main__ - INFO - 📈 Epoch progress: 0.46/6 (7.6%) |
| 2025-11-23 04:43:43,873 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.8%) |
| 2025-11-23 04:44:04,296 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.9%) |
| 2025-11-23 04:44:24,756 - __main__ - INFO - 📈 Epoch progress: 0.48/6 (8.0%) |
| 2025-11-23 04:44:44,869 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.1%) |
| 2025-11-23 04:45:05,284 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.2%) |
| 2025-11-23 04:45:25,479 - __main__ - INFO - 📈 Epoch progress: 0.50/6 (8.4%) |
| 2025-11-23 04:45:46,062 - __main__ - INFO - 📈 Epoch progress: 0.51/6 (8.5%) |
| 2025-11-23 04:46:06,580 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.6%) |
| 2025-11-23 04:46:27,033 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.7%) |
| 2025-11-23 04:46:47,558 - __main__ - INFO - 📈 Epoch progress: 0.53/6 (8.8%) |
| 2025-11-23 04:47:08,065 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.0%) |
| 2025-11-23 04:47:28,658 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.1%) |
| 2025-11-23 04:47:49,153 - __main__ - INFO - 📈 Epoch progress: 0.55/6 (9.2%) |
| 2025-11-23 04:48:10,185 - __main__ - INFO - 📈 Epoch progress: 0.56/6 (9.3%) |
| 2025-11-23 04:48:30,182 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.4%) |
| 2025-11-23 04:48:50,420 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.6%) |
| 2025-11-23 04:49:36,565 - __main__ - INFO - 📊 Evaluation 5: BLEU = 0.1175 |
| 2025-11-23 04:49:36,883 - __main__ - INFO - 📊 Evaluation at step 8000 |
| 2025-11-23 04:49:40,471 - __main__ - INFO - |
| 📤 Pushing checkpoint 8000 to HuggingFace Hub... |
| 2025-11-23 04:49:40,472 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 04:49:52,284 - __main__ - INFO - ✅ Checkpoint 8000 pushed successfully! |
| 2025-11-23 04:49:52,285 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-8000 |
| 2025-11-23 04:50:12,334 - __main__ - INFO - 📈 Epoch progress: 0.58/6 (9.7%) |
| 2025-11-23 04:50:32,901 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.8%) |
| 2025-11-23 04:50:53,451 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.9%) |
| 2025-11-23 04:51:13,926 - __main__ - INFO - 📈 Epoch progress: 0.60/6 (10.0%) |
| 2025-11-23 04:51:33,994 - __main__ - INFO - 📈 Epoch progress: 0.61/6 (10.2%) |
| 2025-11-23 04:51:54,443 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.3%) |
| 2025-11-23 04:52:06,094 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 04:52:16,233 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.4%) |
| 2025-11-23 04:52:36,718 - __main__ - INFO - 📈 Epoch progress: 0.63/6 (10.5%) |
| 2025-11-23 04:52:57,184 - __main__ - INFO - 📈 Epoch progress: 0.64/6 (10.6%) |
| 2025-11-23 04:53:18,002 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.8%) |
| 2025-11-23 04:53:38,479 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.9%) |
| 2025-11-23 04:53:58,840 - __main__ - INFO - 📈 Epoch progress: 0.66/6 (11.0%) |
| 2025-11-23 04:54:19,816 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.1%) |
| 2025-11-23 04:54:40,670 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.2%) |
| 2025-11-23 04:55:00,739 - __main__ - INFO - 📈 Epoch progress: 0.68/6 (11.3%) |
| 2025-11-23 04:55:21,422 - __main__ - INFO - 📈 Epoch progress: 0.69/6 (11.5%) |
| 2025-11-23 04:55:42,073 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.6%) |
| 2025-11-23 04:56:02,920 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.7%) |
| 2025-11-23 04:56:23,022 - __main__ - INFO - 📈 Epoch progress: 0.71/6 (11.8%) |
| 2025-11-23 04:56:44,055 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (11.9%) |
| 2025-11-23 04:57:32,881 - __main__ - INFO - 📊 Evaluation 6: BLEU = 0.1201 |
| 2025-11-23 04:57:33,225 - __main__ - INFO - 📊 Evaluation at step 10000 |
| 2025-11-23 04:57:36,863 - __main__ - INFO - |
| 📤 Pushing checkpoint 10000 to HuggingFace Hub... |
| 2025-11-23 04:57:36,864 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 04:57:46,391 - __main__ - INFO - ✅ Checkpoint 10000 pushed successfully! |
| 2025-11-23 04:57:46,392 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-10000 |
| 2025-11-23 04:58:07,139 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (12.1%) |
| 2025-11-23 04:58:27,867 - __main__ - INFO - 📈 Epoch progress: 0.73/6 (12.2%) |
| 2025-11-23 04:58:48,310 - __main__ - INFO - 📈 Epoch progress: 0.74/6 (12.3%) |
| 2025-11-23 04:59:08,967 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.4%) |
| 2025-11-23 04:59:29,328 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.5%) |
| 2025-11-23 04:59:50,033 - __main__ - INFO - 📈 Epoch progress: 0.76/6 (12.7%) |
| 2025-11-23 05:00:10,362 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.8%) |
| 2025-11-23 05:00:31,242 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.9%) |
| 2025-11-23 05:00:52,420 - __main__ - INFO - 📈 Epoch progress: 0.78/6 (13.0%) |
| 2025-11-23 05:01:13,259 - __main__ - INFO - 📈 Epoch progress: 0.79/6 (13.1%) |
| 2025-11-23 05:01:33,648 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.3%) |
| 2025-11-23 05:01:53,923 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.4%) |
| 2025-11-23 05:02:14,830 - __main__ - INFO - 📈 Epoch progress: 0.81/6 (13.5%) |
| 2025-11-23 05:02:35,660 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.6%) |
| 2025-11-23 05:02:55,972 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.7%) |
| 2025-11-23 05:03:16,924 - __main__ - INFO - 📈 Epoch progress: 0.83/6 (13.9%) |
| 2025-11-23 05:03:37,496 - __main__ - INFO - 📈 Epoch progress: 0.84/6 (14.0%) |
| 2025-11-23 05:03:57,884 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.1%) |
| 2025-11-23 05:04:18,312 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.2%) |
| 2025-11-23 05:04:38,492 - __main__ - INFO - 📈 Epoch progress: 0.86/6 (14.3%) |
| 2025-11-23 05:05:32,672 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:05:32,673 - __main__ - INFO - 📊 Evaluation 7: BLEU = 0.1172 |
| 2025-11-23 05:05:33,486 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:05:33,487 - __main__ - INFO - 📊 Evaluation at step 12000 |
| 2025-11-23 05:05:38,273 - __main__ - INFO - |
| 📤 Pushing checkpoint 12000 to HuggingFace Hub... |
| 2025-11-23 05:05:38,274 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 05:05:50,516 - __main__ - INFO - ✅ Checkpoint 12000 pushed successfully! |
| 2025-11-23 05:05:50,517 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-12000 |
| 2025-11-23 05:06:10,571 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.5%) |
| 2025-11-23 05:06:30,760 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.6%) |
| 2025-11-23 05:06:51,435 - __main__ - INFO - 📈 Epoch progress: 0.88/6 (14.7%) |
| 2025-11-23 05:07:11,929 - __main__ - INFO - 📈 Epoch progress: 0.89/6 (14.8%) |
| 2025-11-23 05:07:32,329 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (14.9%) |
| 2025-11-23 05:07:52,931 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (15.1%) |
| 2025-11-23 05:08:13,396 - __main__ - INFO - 📈 Epoch progress: 0.91/6 (15.2%) |
| 2025-11-23 05:08:34,125 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.3%) |
| 2025-11-23 05:08:54,743 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.4%) |
| 2025-11-23 05:09:15,545 - __main__ - INFO - 📈 Epoch progress: 0.93/6 (15.5%) |
| 2025-11-23 05:09:35,761 - __main__ - INFO - 📈 Epoch progress: 0.94/6 (15.6%) |
| 2025-11-23 05:09:56,678 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.8%) |
| 2025-11-23 05:10:17,145 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.9%) |
| 2025-11-23 05:10:37,457 - __main__ - INFO - 📈 Epoch progress: 0.96/6 (16.0%) |
| 2025-11-23 05:10:57,757 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.1%) |
| 2025-11-23 05:11:18,234 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.2%) |
| 2025-11-23 05:11:39,210 - __main__ - INFO - 📈 Epoch progress: 0.98/6 (16.4%) |
| 2025-11-23 05:11:59,688 - __main__ - INFO - 📈 Epoch progress: 0.99/6 (16.5%) |
| 2025-11-23 05:12:19,771 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.6%) |
| 2025-11-23 05:12:41,806 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.7%) |
| 2025-11-23 05:13:28,678 - __main__ - INFO - 📊 Evaluation 8: BLEU = 0.1178 |
| 2025-11-23 05:13:29,017 - __main__ - INFO - 📊 Evaluation at step 14000 |
| 2025-11-23 05:13:33,956 - __main__ - INFO - |
| 📤 Pushing checkpoint 14000 to HuggingFace Hub... |
| 2025-11-23 05:13:33,956 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 05:13:44,412 - __main__ - INFO - ✅ Checkpoint 14000 pushed successfully! |
| 2025-11-23 05:13:44,414 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-14000 |
| 2025-11-23 05:14:04,241 - __main__ - INFO - 📈 Epoch progress: 1.01/6 (16.8%) |
| 2025-11-23 05:14:24,517 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.0%) |
| 2025-11-23 05:14:44,594 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.1%) |
| 2025-11-23 05:15:05,254 - __main__ - INFO - 📈 Epoch progress: 1.03/6 (17.2%) |
| 2025-11-23 05:15:25,675 - __main__ - INFO - 📈 Epoch progress: 1.04/6 (17.3%) |
| 2025-11-23 05:15:46,194 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.4%) |
| 2025-11-23 05:16:06,354 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.6%) |
| 2025-11-23 05:16:26,813 - __main__ - INFO - 📈 Epoch progress: 1.06/6 (17.7%) |
| 2025-11-23 05:16:47,591 - __main__ - INFO - 📈 Epoch progress: 1.07/6 (17.8%) |
| 2025-11-23 05:17:08,129 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (17.9%) |
| 2025-11-23 05:17:28,440 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (18.0%) |
| 2025-11-23 05:17:48,715 - __main__ - INFO - 📈 Epoch progress: 1.09/6 (18.2%) |
| 2025-11-23 05:18:09,243 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.3%) |
| 2025-11-23 05:18:29,913 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.4%) |
| 2025-11-23 05:18:51,203 - __main__ - INFO - 📈 Epoch progress: 1.11/6 (18.5%) |
| 2025-11-23 05:19:11,745 - __main__ - INFO - 📈 Epoch progress: 1.12/6 (18.6%) |
| 2025-11-23 05:19:32,497 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.8%) |
| 2025-11-23 05:19:53,187 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.9%) |
| 2025-11-23 05:20:13,965 - __main__ - INFO - 📈 Epoch progress: 1.14/6 (19.0%) |
| 2025-11-23 05:20:34,523 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.1%) |
| 2025-11-23 05:21:24,032 - __main__ - INFO - 📊 Evaluation 9: BLEU = 0.1183 |
| 2025-11-23 05:21:24,377 - __main__ - INFO - 📊 Evaluation at step 16000 |
| 2025-11-23 05:21:28,472 - __main__ - INFO - |
| 📤 Pushing checkpoint 16000 to HuggingFace Hub... |
| 2025-11-23 05:21:28,473 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 05:21:40,266 - __main__ - INFO - ✅ Checkpoint 16000 pushed successfully! |
| 2025-11-23 05:21:40,267 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-16000 |
| 2025-11-23 05:22:00,311 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.2%) |
| 2025-11-23 05:22:21,476 - __main__ - INFO - 📈 Epoch progress: 1.16/6 (19.4%) |
| 2025-11-23 05:22:42,000 - __main__ - INFO - 📈 Epoch progress: 1.17/6 (19.5%) |
| 2025-11-23 05:23:02,752 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.6%) |
| 2025-11-23 05:23:23,438 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.7%) |
| 2025-11-23 05:23:44,156 - __main__ - INFO - 📈 Epoch progress: 1.19/6 (19.8%) |
| 2025-11-23 05:24:04,702 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (19.9%) |
| 2025-11-23 05:24:25,167 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (20.1%) |
| 2025-11-23 05:24:45,698 - __main__ - INFO - 📈 Epoch progress: 1.21/6 (20.2%) |
| 2025-11-23 05:25:06,038 - __main__ - INFO - 📈 Epoch progress: 1.22/6 (20.3%) |
| 2025-11-23 05:25:26,872 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.4%) |
| 2025-11-23 05:25:47,382 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.5%) |
| 2025-11-23 05:26:07,913 - __main__ - INFO - 📈 Epoch progress: 1.24/6 (20.7%) |
| 2025-11-23 05:26:28,562 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.8%) |
| 2025-11-23 05:26:49,511 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.9%) |
| 2025-11-23 05:27:10,199 - __main__ - INFO - 📈 Epoch progress: 1.26/6 (21.0%) |
| 2025-11-23 05:27:30,936 - __main__ - INFO - 📈 Epoch progress: 1.27/6 (21.1%) |
| 2025-11-23 05:27:51,405 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.3%) |
| 2025-11-23 05:28:11,636 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.4%) |
| 2025-11-23 05:28:32,412 - __main__ - INFO - 📈 Epoch progress: 1.29/6 (21.5%) |
| 2025-11-23 05:29:18,518 - __main__ - INFO - 📊 Evaluation 10: BLEU = 0.1219 |
| 2025-11-23 05:29:18,888 - __main__ - INFO - 📊 Evaluation at step 18000 |
| 2025-11-23 05:29:22,587 - __main__ - INFO - |
| 📤 Pushing checkpoint 18000 to HuggingFace Hub... |
| 2025-11-23 05:29:22,588 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 05:29:32,167 - __main__ - INFO - ✅ Checkpoint 18000 pushed successfully! |
| 2025-11-23 05:29:32,168 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-18000 |
| 2025-11-23 05:29:52,949 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.6%) |
| 2025-11-23 05:30:13,314 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.7%) |
| 2025-11-23 05:30:33,392 - __main__ - INFO - 📈 Epoch progress: 1.31/6 (21.9%) |
| 2025-11-23 05:30:53,705 - __main__ - INFO - 📈 Epoch progress: 1.32/6 (22.0%) |
| 2025-11-23 05:31:22,316 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.1%) |
| 2025-11-23 05:31:43,345 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.2%) |
| 2025-11-23 05:32:03,977 - __main__ - INFO - 📈 Epoch progress: 1.34/6 (22.3%) |
| 2025-11-23 05:32:24,473 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.5%) |
| 2025-11-23 05:32:45,183 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.6%) |
| 2025-11-23 05:33:05,536 - __main__ - INFO - 📈 Epoch progress: 1.36/6 (22.7%) |
| 2025-11-23 05:33:26,144 - __main__ - INFO - 📈 Epoch progress: 1.37/6 (22.8%) |
| 2025-11-23 05:33:46,767 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (22.9%) |
| 2025-11-23 05:34:07,141 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (23.1%) |
| 2025-11-23 05:34:27,441 - __main__ - INFO - 📈 Epoch progress: 1.39/6 (23.2%) |
| 2025-11-23 05:34:48,175 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.3%) |
| 2025-11-23 05:35:08,694 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.4%) |
| 2025-11-23 05:35:29,100 - __main__ - INFO - 📈 Epoch progress: 1.41/6 (23.5%) |
| 2025-11-23 05:35:49,987 - __main__ - INFO - 📈 Epoch progress: 1.42/6 (23.7%) |
| 2025-11-23 05:36:10,311 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.8%) |
| 2025-11-23 05:36:30,472 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.9%) |
| 2025-11-23 05:37:17,929 - __main__ - INFO - 📊 Evaluation 11: BLEU = 0.1225 |
| 2025-11-23 05:37:18,295 - __main__ - INFO - 📊 Evaluation at step 20000 |
| 2025-11-23 05:37:22,372 - __main__ - INFO - |
| 📤 Pushing checkpoint 20000 to HuggingFace Hub... |
| 2025-11-23 05:37:22,373 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 05:37:33,166 - __main__ - INFO - ✅ Checkpoint 20000 pushed successfully! |
| 2025-11-23 05:37:33,167 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-20000 |
| 2025-11-23 05:37:54,117 - __main__ - INFO - 📈 Epoch progress: 1.44/6 (24.0%) |
| 2025-11-23 05:38:14,559 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.1%) |
| 2025-11-23 05:38:35,292 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.2%) |
| 2025-11-23 05:38:55,844 - __main__ - INFO - 📈 Epoch progress: 1.46/6 (24.4%) |
| 2025-11-23 05:39:17,013 - __main__ - INFO - 📈 Epoch progress: 1.47/6 (24.5%) |
| 2025-11-23 05:39:37,798 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.6%) |
| 2025-11-23 05:39:58,948 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.7%) |
| 2025-11-23 05:40:19,537 - __main__ - INFO - 📈 Epoch progress: 1.49/6 (24.8%) |
| 2025-11-23 05:40:40,399 - __main__ - INFO - 📈 Epoch progress: 1.50/6 (25.0%) |
| 2025-11-23 05:41:00,822 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.1%) |
| 2025-11-23 05:41:21,593 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.2%) |
| 2025-11-23 05:41:42,212 - __main__ - INFO - 📈 Epoch progress: 1.52/6 (25.3%) |
| 2025-11-23 05:42:03,403 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.4%) |
| 2025-11-23 05:42:23,842 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.6%) |
| 2025-11-23 05:42:44,097 - __main__ - INFO - 📈 Epoch progress: 1.54/6 (25.7%) |
| 2025-11-23 05:43:04,918 - __main__ - INFO - 📈 Epoch progress: 1.55/6 (25.8%) |
| 2025-11-23 05:43:25,164 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (25.9%) |
| 2025-11-23 05:43:54,430 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:43:54,432 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (26.0%) |
| 2025-11-23 05:43:54,742 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:44:14,598 - __main__ - INFO - 📈 Epoch progress: 1.57/6 (26.2%) |
| 2025-11-23 05:44:35,347 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.3%) |
| 2025-11-23 05:45:31,338 - __main__ - INFO - 📊 Evaluation 12: BLEU = 0.1133 |
| 2025-11-23 05:45:31,674 - __main__ - INFO - 📊 Evaluation at step 22000 |
| 2025-11-23 05:45:35,719 - __main__ - INFO - |
| 📤 Pushing checkpoint 22000 to HuggingFace Hub... |
| 2025-11-23 05:45:35,721 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 05:45:44,938 - __main__ - INFO - ✅ Checkpoint 22000 pushed successfully! |
| 2025-11-23 05:45:44,939 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-22000 |
| 2025-11-23 05:46:05,000 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.4%) |
| 2025-11-23 05:46:25,834 - __main__ - INFO - 📈 Epoch progress: 1.59/6 (26.5%) |
| 2025-11-23 05:46:46,517 - __main__ - INFO - 📈 Epoch progress: 1.60/6 (26.6%) |
| 2025-11-23 05:47:06,755 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.8%) |
| 2025-11-23 05:47:27,574 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.9%) |
| 2025-11-23 05:47:47,894 - __main__ - INFO - 📈 Epoch progress: 1.62/6 (27.0%) |
| 2025-11-23 05:48:08,503 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.1%) |
| 2025-11-23 05:48:29,258 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.2%) |
| 2025-11-23 05:48:50,206 - __main__ - INFO - 📈 Epoch progress: 1.64/6 (27.4%) |
| 2025-11-23 05:49:11,012 - __main__ - INFO - 📈 Epoch progress: 1.65/6 (27.5%) |
| 2025-11-23 05:49:22,154 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:49:32,090 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:49:32,091 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.6%) |
| 2025-11-23 05:49:32,420 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:49:43,023 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:49:52,768 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.7%) |
| 2025-11-23 05:50:13,180 - __main__ - INFO - 📈 Epoch progress: 1.67/6 (27.8%) |
| 2025-11-23 05:50:33,663 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.0%) |
| 2025-11-23 05:50:54,100 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.1%) |
| 2025-11-23 05:51:14,972 - __main__ - INFO - 📈 Epoch progress: 1.69/6 (28.2%) |
| 2025-11-23 05:51:35,547 - __main__ - INFO - 📈 Epoch progress: 1.70/6 (28.3%) |
| 2025-11-23 05:51:55,721 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.4%) |
| 2025-11-23 05:52:16,479 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.5%) |
| 2025-11-23 05:52:36,928 - __main__ - INFO - 📈 Epoch progress: 1.72/6 (28.7%) |
| 2025-11-23 05:53:20,026 - __main__ - INFO - 📊 Evaluation 13: BLEU = 0.1197 |
| 2025-11-23 05:53:20,363 - __main__ - INFO - 📊 Evaluation at step 24000 |
| 2025-11-23 05:53:24,050 - __main__ - INFO - |
| 📤 Pushing checkpoint 24000 to HuggingFace Hub... |
| 2025-11-23 05:53:24,050 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 05:53:35,770 - __main__ - INFO - ✅ Checkpoint 24000 pushed successfully! |
| 2025-11-23 05:53:35,771 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-24000 |
| 2025-11-23 05:53:55,952 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.8%) |
| 2025-11-23 05:54:16,149 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.9%) |
| 2025-11-23 05:54:36,530 - __main__ - INFO - 📈 Epoch progress: 1.74/6 (29.0%) |
| 2025-11-23 05:54:57,769 - __main__ - INFO - 📈 Epoch progress: 1.75/6 (29.1%) |
| 2025-11-23 05:55:17,644 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.3%) |
| 2025-11-23 05:55:38,341 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.4%) |
| 2025-11-23 05:55:58,622 - __main__ - INFO - 📈 Epoch progress: 1.77/6 (29.5%) |
| 2025-11-23 05:56:18,961 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.6%) |
| 2025-11-23 05:56:39,998 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.7%) |
| 2025-11-23 05:57:00,353 - __main__ - INFO - 📈 Epoch progress: 1.79/6 (29.9%) |
| 2025-11-23 05:57:20,656 - __main__ - INFO - 📈 Epoch progress: 1.80/6 (30.0%) |
| 2025-11-23 05:57:36,826 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:57:46,733 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:57:46,734 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.1%) |
| 2025-11-23 05:57:47,030 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:57:56,836 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:58:06,747 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:58:06,747 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.2%) |
| 2025-11-23 05:58:07,049 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:58:16,831 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 05:58:27,177 - __main__ - INFO - 📈 Epoch progress: 1.82/6 (30.3%) |
| 2025-11-23 05:58:48,335 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.5%) |
| 2025-11-23 05:59:09,475 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.6%) |
| 2025-11-23 05:59:29,797 - __main__ - INFO - 📈 Epoch progress: 1.84/6 (30.7%) |
| 2025-11-23 05:59:50,498 - __main__ - INFO - 📈 Epoch progress: 1.85/6 (30.8%) |
| 2025-11-23 06:00:11,935 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (30.9%) |
| 2025-11-23 06:00:32,557 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (31.1%) |
| 2025-11-23 06:01:16,076 - __main__ - INFO - 📊 Evaluation 14: BLEU = 0.1220 |
| 2025-11-23 06:01:16,758 - __main__ - INFO - 📊 Evaluation at step 26000 |
| 2025-11-23 06:01:20,119 - __main__ - INFO - |
| 📤 Pushing checkpoint 26000 to HuggingFace Hub... |
| 2025-11-23 06:01:20,120 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 06:01:29,811 - __main__ - INFO - ✅ Checkpoint 26000 pushed successfully! |
| 2025-11-23 06:01:29,814 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-26000 |
| 2025-11-23 06:01:49,787 - __main__ - INFO - 📈 Epoch progress: 1.87/6 (31.2%) |
| 2025-11-23 06:02:10,995 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.3%) |
| 2025-11-23 06:02:31,913 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.4%) |
| 2025-11-23 06:02:53,351 - __main__ - INFO - 📈 Epoch progress: 1.89/6 (31.5%) |
| 2025-11-23 06:03:13,904 - __main__ - INFO - 📈 Epoch progress: 1.90/6 (31.7%) |
| 2025-11-23 06:03:35,786 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.8%) |
| 2025-11-23 06:03:55,853 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.9%) |
| 2025-11-23 06:04:16,517 - __main__ - INFO - 📈 Epoch progress: 1.92/6 (32.0%) |
| 2025-11-23 06:04:37,380 - __main__ - INFO - 📈 Epoch progress: 1.93/6 (32.1%) |
| 2025-11-23 06:05:08,205 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.3%) |
| 2025-11-23 06:05:28,854 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.4%) |
| 2025-11-23 06:05:49,622 - __main__ - INFO - 📈 Epoch progress: 1.95/6 (32.5%) |
| 2025-11-23 06:06:10,932 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.6%) |
| 2025-11-23 06:06:31,737 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.7%) |
| 2025-11-23 06:06:51,966 - __main__ - INFO - 📈 Epoch progress: 1.97/6 (32.8%) |
| 2025-11-23 06:07:12,494 - __main__ - INFO - 📈 Epoch progress: 1.98/6 (33.0%) |
| 2025-11-23 06:07:33,444 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.1%) |
| 2025-11-23 06:07:55,079 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.2%) |
| 2025-11-23 06:08:15,763 - __main__ - INFO - 📈 Epoch progress: 2.00/6 (33.3%) |
| 2025-11-23 06:08:37,725 - __main__ - INFO - 📈 Epoch progress: 2.01/6 (33.4%) |
| 2025-11-23 06:09:16,114 - __main__ - INFO - 📊 Evaluation 15: BLEU = 0.1212 |
| 2025-11-23 06:09:16,813 - __main__ - INFO - 📊 Evaluation at step 28000 |
| 2025-11-23 06:09:20,719 - __main__ - INFO - |
| 📤 Pushing checkpoint 28000 to HuggingFace Hub... |
| 2025-11-23 06:09:20,719 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 06:09:30,705 - __main__ - INFO - ✅ Checkpoint 28000 pushed successfully! |
| 2025-11-23 06:09:30,717 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-28000 |
| 2025-11-23 06:09:34,395 - __main__ - INFO - ✅ MLflow: Training finished - status_code=2.0 (FINISHED) |
| 2025-11-23 06:09:34,733 - __main__ - INFO - |
| 💾 Saving model to models/finetuned-sw-en-phase2 |
| 2025-11-23 06:09:36,180 - __main__ - INFO - ✅ Training complete! |
| 2025-11-23 06:09:36,181 - __main__ - INFO - |
| 📊 STEP 3: Final Evaluation |
| 2025-11-23 06:09:36,530 - __main__ - INFO - ✅ MLflow: Final evaluation starting - status_code=3.0 |
| 2025-11-23 06:09:36,531 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 06:09:36,532 - __main__ - INFO - 📊 Running FINAL evaluation on test set... |
| 2025-11-23 06:09:36,533 - __main__ - INFO - ====================================================================== |
| 2025-11-23 06:09:36,534 - __main__ - INFO - Tokenizing test dataset... |
| 2025-11-23 06:09:38,520 - datasets.fingerprint - WARNING - Parameter 'function'=<bound method TranslationTrainer.preprocess_function of <__main__.TranslationTrainer object at 0x7b18d516be60>> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown. |
| 2025-11-23 06:41:24,155 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-23 06:41:24,162 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-23 06:41:24,163 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-23 06:41:24,163 - __main__ - INFO - Model output: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-23 06:41:24,166 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-23 06:41:24,166 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-23 06:41:24,167 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 06:41:24,167 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-23 06:41:24,168 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-23 06:41:24,168 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-23 06:41:24,169 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-23 06:41:24,170 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-23 06:41:24,178 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-23 06:41:24,179 - __main__ - INFO - ✅ MLflow tracking URI set: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 06:41:24,827 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-23 06:41:25,688 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251123_064125 |
| 2025-11-23 06:41:25,690 - __main__ - INFO - Run ID: 54a97666ebd34e46a8ce61941ee6341c |
| 2025-11-23 06:41:25,691 - __main__ - INFO - Tracking URI: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 06:41:26,048 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow |
| 2025-11-23 06:41:26,051 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-23 06:41:26,061 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-23 06:41:26,061 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-23 06:41:26,062 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-23 06:41:31,514 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-23 06:41:31,515 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-23 06:41:31,921 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-23 06:41:32,192 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-23 06:41:32,193 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-23 06:41:33,774 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-23 06:41:36,713 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:39,276 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:41,691 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:44,079 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:46,461 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:48,840 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:51,248 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:53,590 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:55,986 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-23 06:41:58,421 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:00,853 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:03,291 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:05,726 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:08,157 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:10,577 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:12,974 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:15,394 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:17,840 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:20,309 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:22,760 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:25,196 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:27,644 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:30,074 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:32,478 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:34,882 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-23 06:42:35,119 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-23 06:42:35,156 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-23 06:42:36,942 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-23 06:42:39,956 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-23 06:42:42,426 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-23 06:42:44,876 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-23 06:42:47,303 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-23 06:42:49,779 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-23 06:42:52,232 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-23 06:42:54,688 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-23 06:42:57,144 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-23 06:42:59,590 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-23 06:43:02,033 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-23 06:43:04,493 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 |
| 2025-11-23 06:43:06,941 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 |
| 2025-11-23 06:43:09,378 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 |
| 2025-11-23 06:43:11,817 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 |
| 2025-11-23 06:43:14,265 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 |
| 2025-11-23 06:43:16,716 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 |
| 2025-11-23 06:43:18,994 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) |
| 2025-11-23 06:43:19,040 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 |
| 2025-11-23 06:43:19,041 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-23 06:43:19,042 - __main__ - INFO - Sample 1: |
| 2025-11-23 06:43:19,043 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... |
| 2025-11-23 06:43:19,044 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... |
| 2025-11-23 06:43:19,045 - __main__ - INFO - Sample 2: |
| 2025-11-23 06:43:19,046 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... |
| 2025-11-23 06:43:19,046 - __main__ - INFO - Target (6 words): Job had said no such thing.... |
| 2025-11-23 06:43:19,047 - __main__ - INFO - Sample 3: |
| 2025-11-23 06:43:19,048 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... |
| 2025-11-23 06:43:19,048 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... |
| 2025-11-23 06:43:24,401 - __main__ - INFO - Filtering statistics: {'kept': 4202243} |
| 2025-11-23 06:43:24,417 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 |
| 2025-11-23 06:43:28,817 - __main__ - INFO - Removed 518 duplicates |
| 2025-11-23 06:43:29,060 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 |
| 2025-11-23 06:43:35,738 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 06:43:35,740 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-23 06:43:35,740 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-23 06:43:35,741 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-23 06:43:35,742 - __main__ - INFO - Test: 210,087 samples |
| 2025-11-23 06:43:35,743 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 06:43:36,203 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-23 06:46:40,471 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-23 06:46:40,473 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-23 06:46:40,474 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-23 06:46:40,475 - __main__ - INFO - Test: 210,087 samples |
|
|
| 2025-11-23 06:46:41,873 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) |
| 2025-11-23 06:46:41,874 - __main__ - INFO - Skipping tokenization - this saves significant time! |
| 2025-11-23 06:46:41,875 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... |
| 2025-11-23 06:47:24,470 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-23 06:47:24,472 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-23 06:47:24,839 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-23 06:47:25,934 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-23 06:47:32,709 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-23 06:47:33,265 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-23 06:47:33,266 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-23 06:47:33,298 - __main__ - INFO - 📊 Limiting evaluation dataset from 420225 to 2000 samples |
| 2025-11-23 06:47:33,335 - __main__ - INFO - ✅ Trainer initialized |
| 2025-11-23 06:47:33,336 - __main__ - INFO - |
| 🔬 STEP 1: Baseline Evaluation |
| 2025-11-23 06:47:33,836 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 |
| 2025-11-23 06:47:33,837 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 06:47:33,838 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... |
| 2025-11-23 06:47:33,839 - __main__ - INFO - ====================================================================== |
| 2025-11-23 06:48:23,063 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1215 |
| 2025-11-23 06:48:23,065 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-23 06:48:23,106 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... |
| 2025-11-23 06:48:23,115 - __main__ - INFO - Batch 1/16... |
| 2025-11-23 06:48:25,740 - __main__ - INFO - Batch 5/16... |
| 2025-11-23 06:48:27,878 - __main__ - INFO - Batch 10/16... |
| 2025-11-23 06:48:30,442 - __main__ - INFO - Batch 15/16... |
| 2025-11-23 06:48:31,207 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-23 06:48:31,228 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-23 06:48:31,237 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-23 06:48:31,238 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-23 06:48:31,239 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision |
| 2025-11-23 06:48:31,241 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-23 06:48:36,590 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 |
| 2025-11-23 06:48:36,591 - __main__ - INFO - |
| ✅ Baseline Results (Test Set): |
| 2025-11-23 06:48:36,592 - __main__ - INFO - BLEU: 0.1215 |
| 2025-11-23 06:48:36,592 - __main__ - INFO - chrF: 32.36 |
| 2025-11-23 06:48:36,593 - __main__ - INFO - COMET: 0.1977 |
| 2025-11-23 06:48:36,594 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 06:48:36,595 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 06:48:36,596 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-23 06:48:36,596 - __main__ - INFO - ====================================================================== |
| 2025-11-23 06:48:36,597 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-23 06:48:36,599 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-23 06:48:38,317 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-23 06:48:38,378 - __main__ - INFO - ✅ Domain chrF: 1.22 |
| 2025-11-23 06:48:38,380 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-23 06:48:38,386 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-23 06:48:38,394 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-23 06:48:38,395 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-23 06:48:38,397 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-23 06:48:41,106 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 |
| 2025-11-23 06:48:41,115 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-23 06:48:41,116 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-23 06:48:41,117 - __main__ - INFO - chrF: 1.22 |
| 2025-11-23 06:48:41,118 - __main__ - INFO - COMET-QE: 0.0001 |
| 2025-11-23 06:48:41,120 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 06:48:41,726 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 |
| 2025-11-23 06:48:41,737 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json |
| 2025-11-23 06:48:41,742 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics |
| 2025-11-23 06:48:41,744 - __main__ - INFO - |
| 🚀 STEP 2: Fine-tuning |
| 2025-11-23 06:48:41,745 - __main__ - INFO - Metrics are being logged to MLflow in real-time |
| 2025-11-23 06:48:41,746 - __main__ - INFO - MLflow URL: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 06:48:41,747 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs |
| 2025-11-23 06:48:41,749 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') |
| 2025-11-23 06:48:41,750 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 06:48:41,751 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 06:48:41,752 - __main__ - INFO - 🚀 Starting training... |
| 2025-11-23 06:48:41,753 - __main__ - INFO - ====================================================================== |
| 2025-11-23 06:48:42,779 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) |
| 2025-11-23 06:49:20,694 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.1%) |
| 2025-11-23 06:49:56,400 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.2%) |
| 2025-11-23 06:50:32,830 - __main__ - INFO - 📈 Epoch progress: 0.02/6 (0.4%) |
| 2025-11-23 06:51:08,804 - __main__ - INFO - 📈 Epoch progress: 0.03/6 (0.5%) |
| 2025-11-23 06:51:44,666 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.6%) |
| 2025-11-23 06:52:20,882 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.7%) |
| 2025-11-23 06:52:59,996 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 06:52:59,998 - __main__ - INFO - 📈 Epoch progress: 0.05/6 (0.8%) |
| 2025-11-23 06:53:00,318 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 06:53:17,708 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 06:53:35,669 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.0%) |
| 2025-11-23 06:54:11,635 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.1%) |
| 2025-11-23 06:54:47,036 - __main__ - INFO - 📈 Epoch progress: 0.07/6 (1.2%) |
| 2025-11-23 06:55:22,872 - __main__ - INFO - 📈 Epoch progress: 0.08/6 (1.3%) |
| 2025-11-23 06:55:57,782 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.4%) |
| 2025-11-23 06:56:34,083 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.6%) |
| 2025-11-23 06:57:10,153 - __main__ - INFO - 📈 Epoch progress: 0.10/6 (1.7%) |
| 2025-11-23 06:57:45,970 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.8%) |
| 2025-11-23 06:58:21,543 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 06:58:21,545 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.9%) |
| 2025-11-23 06:58:21,879 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 06:58:57,226 - __main__ - INFO - 📈 Epoch progress: 0.12/6 (2.0%) |
| 2025-11-23 06:59:32,715 - __main__ - INFO - 📈 Epoch progress: 0.13/6 (2.2%) |
| 2025-11-23 07:00:08,615 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.3%) |
| 2025-11-23 07:00:45,098 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.4%) |
| 2025-11-23 07:02:10,816 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1169 |
| 2025-11-23 07:02:11,162 - __main__ - INFO - 📊 Evaluation at step 2000 |
| 2025-11-23 07:02:15,403 - __main__ - INFO - ✅ Repository brendaogutu/mul-sw-en-translation-phase2 ready |
| 2025-11-23 07:02:15,420 - __main__ - INFO - |
| 📤 Pushing checkpoint 2000 to HuggingFace Hub... |
| 2025-11-23 07:02:15,421 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 07:02:30,795 - __main__ - INFO - ✅ Checkpoint 2000 pushed successfully! |
| 2025-11-23 07:02:30,800 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-2000 |
| 2025-11-23 07:03:07,071 - __main__ - INFO - 📈 Epoch progress: 0.15/6 (2.5%) |
| 2025-11-23 07:03:43,908 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.6%) |
| 2025-11-23 07:04:20,289 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.7%) |
| 2025-11-23 07:04:56,501 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:04:56,516 - __main__ - INFO - 📈 Epoch progress: 0.17/6 (2.9%) |
| 2025-11-23 07:04:56,852 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:05:32,222 - __main__ - INFO - 📈 Epoch progress: 0.18/6 (3.0%) |
| 2025-11-23 07:06:08,173 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.1%) |
| 2025-11-23 07:06:43,702 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.2%) |
| 2025-11-23 07:07:20,094 - __main__ - INFO - 📈 Epoch progress: 0.20/6 (3.3%) |
| 2025-11-23 07:07:55,646 - __main__ - INFO - 📈 Epoch progress: 0.21/6 (3.5%) |
| 2025-11-23 07:08:31,361 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.6%) |
| 2025-11-23 07:09:06,938 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.7%) |
| 2025-11-23 07:09:42,535 - __main__ - INFO - 📈 Epoch progress: 0.23/6 (3.8%) |
| 2025-11-23 07:10:19,180 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (3.9%) |
| 2025-11-23 07:10:55,369 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (4.1%) |
| 2025-11-23 07:11:31,824 - __main__ - INFO - 📈 Epoch progress: 0.25/6 (4.2%) |
| 2025-11-23 07:12:08,087 - __main__ - INFO - 📈 Epoch progress: 0.26/6 (4.3%) |
| 2025-11-23 07:12:44,805 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:12:44,816 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.4%) |
| 2025-11-23 07:12:45,137 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:13:21,013 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.5%) |
| 2025-11-23 07:13:55,915 - __main__ - INFO - 📈 Epoch progress: 0.28/6 (4.7%) |
| 2025-11-23 07:14:32,742 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.8%) |
| 2025-11-23 07:15:53,704 - __main__ - INFO - 📊 Evaluation 3: BLEU = 0.1186 |
| 2025-11-23 07:15:54,113 - __main__ - INFO - 📊 Evaluation at step 4000 |
| 2025-11-23 07:15:58,436 - __main__ - INFO - |
| 📤 Pushing checkpoint 4000 to HuggingFace Hub... |
| 2025-11-23 07:15:58,437 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 07:16:05,731 - __main__ - INFO - ✅ Checkpoint 4000 pushed successfully! |
| 2025-11-23 07:16:05,732 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-4000 |
| 2025-11-23 07:16:42,057 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.9%) |
| 2025-11-23 07:17:17,223 - __main__ - INFO - 📈 Epoch progress: 0.30/6 (5.0%) |
| 2025-11-23 07:17:52,523 - __main__ - INFO - 📈 Epoch progress: 0.31/6 (5.1%) |
| 2025-11-23 07:18:28,936 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.3%) |
| 2025-11-23 07:19:05,115 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.4%) |
| 2025-11-23 07:19:40,549 - __main__ - INFO - 📈 Epoch progress: 0.33/6 (5.5%) |
| 2025-11-23 07:20:16,666 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.6%) |
| 2025-11-23 07:20:52,558 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.7%) |
| 2025-11-23 07:21:28,256 - __main__ - INFO - 📈 Epoch progress: 0.35/6 (5.9%) |
| 2025-11-23 07:22:04,280 - __main__ - INFO - 📈 Epoch progress: 0.36/6 (6.0%) |
| 2025-11-23 07:22:40,312 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.1%) |
| 2025-11-23 07:23:15,817 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.2%) |
| 2025-11-23 07:23:51,373 - __main__ - INFO - 📈 Epoch progress: 0.38/6 (6.3%) |
| 2025-11-23 07:24:28,475 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.5%) |
| 2025-11-23 07:25:05,401 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.6%) |
| 2025-11-23 07:25:41,747 - __main__ - INFO - 📈 Epoch progress: 0.40/6 (6.7%) |
| 2025-11-23 07:26:17,592 - __main__ - INFO - 📈 Epoch progress: 0.41/6 (6.8%) |
| 2025-11-23 07:26:53,822 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (6.9%) |
| 2025-11-23 07:27:29,503 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (7.0%) |
| 2025-11-23 07:28:04,934 - __main__ - INFO - 📈 Epoch progress: 0.43/6 (7.2%) |
| 2025-11-23 07:29:35,171 - __main__ - INFO - 📊 Evaluation 4: BLEU = 0.1098 |
| 2025-11-23 07:29:35,497 - __main__ - INFO - 📊 Evaluation at step 6000 |
| 2025-11-23 07:29:39,457 - __main__ - INFO - |
| 📤 Pushing checkpoint 6000 to HuggingFace Hub... |
| 2025-11-23 07:29:39,458 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 07:29:47,255 - __main__ - INFO - ✅ Checkpoint 6000 pushed successfully! |
| 2025-11-23 07:29:47,257 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-6000 |
| 2025-11-23 07:30:22,257 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.3%) |
| 2025-11-23 07:30:58,369 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.4%) |
| 2025-11-23 07:31:35,126 - __main__ - INFO - 📈 Epoch progress: 0.45/6 (7.5%) |
| 2025-11-23 07:32:12,216 - __main__ - INFO - 📈 Epoch progress: 0.46/6 (7.6%) |
| 2025-11-23 07:32:37,401 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:32:55,033 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:32:55,036 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.8%) |
| 2025-11-23 07:32:55,345 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:33:32,031 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.9%) |
| 2025-11-23 07:34:08,553 - __main__ - INFO - 📈 Epoch progress: 0.48/6 (8.0%) |
| 2025-11-23 07:34:45,333 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.1%) |
| 2025-11-23 07:35:21,813 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.2%) |
| 2025-11-23 07:35:57,659 - __main__ - INFO - 📈 Epoch progress: 0.50/6 (8.4%) |
| 2025-11-23 07:36:34,656 - __main__ - INFO - 📈 Epoch progress: 0.51/6 (8.5%) |
| 2025-11-23 07:37:11,328 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.6%) |
| 2025-11-23 07:37:48,164 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.7%) |
| 2025-11-23 07:38:25,336 - __main__ - INFO - 📈 Epoch progress: 0.53/6 (8.8%) |
| 2025-11-23 07:39:03,247 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.0%) |
| 2025-11-23 07:39:22,404 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:39:40,831 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:39:40,833 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.1%) |
| 2025-11-23 07:39:41,190 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:39:59,010 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '<!DOCTYPE html> |
| <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]--> |
| <!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]--> |
| <!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]--> |
| <head> |
| <title>Cloudflare Tunnel error | food-accepts-blocked-refinance.trycloudflare.com | Cloudflare</title> |
| <meta charset="UTF-8" /> |
| <meta http-equiv="Content-Type" content="text/html |
| <meta http-equiv="X-UA-Compatible" content="IE=Edge" /> |
| <meta name="robots" content="noindex, nofollow" /> |
| <meta name="viewport" content="width=device-width,initial-scale=1" /> |
| <link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" /> |
|
|
|
|
| <script> |
| (function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.stringify){var e=function(a){var c=document.getElementById("error-feedback-survey"),d=document.getElementById("error-feedback-success"),b=new XMLHttpRequest |
| b.send(JSON.stringify(a)) |
| </script> |
|
|
| <script defer src="https://performance.radar.cloudflare.com/beacon.js"></script> |
| </head> |
| <body> |
| <div id="cf-wrapper"> |
| <div class="cf-alert cf-alert-error cf-cookie-error hidden" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div> |
| <div id="cf-error-details" class="p-0"> |
| <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased"> |
| <h1 class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight"> |
| <span data-translate="error">Error</span> |
| <span>1033</span> |
| </h1> |
| <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">Ray ID: 9a2f040dc6afa2c4 &bull |
| <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">2025-11-23 07:39:59 UTC</span> |
| <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">Cloudflare Tunnel error</h2> |
| </header> |
|
|
| <section class="w-240 lg:w-full mx-auto mb-8 lg:px-8"> |
| <div id="what-happened-section" class="w-1/2 md:w-full"> |
| <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_happened">What happened?</h2> |
| <p>You've requested a page on a website (food-accepts-blocked-refinance.trycloudflare.com) that is on the <a href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a> network. The host (food-accepts-blocked-refinance.trycloudflare.com) is configured as a Cloudflare Tunnel, and Cloudflare is currently unable to resolve it. |
| |
| </div> |
|
|
| |
| <div id="resolution-copy-section" class="w-1/2 mt-6 text-15 leading-normal"> |
| <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_can_i_do">What can I do?</h2> |
| <p><strong>If you are a visitor of this website:</strong><br />Please try again in a few minutes.</p><p><strong>If you are the owner of this website:</strong><br />Ensure that cloudflared is running and can reach the network. You may wish to enable <a rel="noopener noreferrer" href="https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/routing-to-tunnel/lb/">load balancing</a> for your tunnel.</p> |
| </div> |
| |
| </section> |
|
|
| <div class="feedback-hidden py-8 text-center" id="error-feedback"> |
| <div id="error-feedback-survey" class="footer-line-wrapper"> |
| Was this page helpful? |
| <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-yes" type="button">Yes</button> |
| <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-no" type="button">No</button> |
| </div> |
| <div class="feedback-success feedback-hidden" id="error-feedback-success"> |
| Thank you for your feedback! |
| </div> |
| </div> |
|
|
|
|
| <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300"> |
| <p class="text-13"> |
| <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">9a2f040dc6afa2c4</strong></span> |
| <span class="cf-footer-separator sm:hidden">&bull |
| <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1"> |
| Your IP: |
| <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button> |
| <span class="hidden" id="cf-footer-ip">69.30.85.10</span> |
| <span class="cf-footer-separator sm:hidden">&bull |
| </span> |
| <span class="cf-footer-item sm:block sm:mb-1"><span>Performance & |
| |
| </p> |
| <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal") |
| </div><!-- /.error-footer --> |
|
|
|
|
| </div><!-- /#cf-error-details --> |
| </div><!-- /#cf-wrapper --> |
|
|
| <script> |
| window._cf_translation = {} |
| |
| |
| </script> |
|
|
| </body> |
| </html> |
| ' |
| 2025-11-23 07:40:16,962 - __main__ - INFO - 📈 Epoch progress: 0.55/6 (9.2%) |
| 2025-11-23 07:40:52,752 - __main__ - INFO - 📈 Epoch progress: 0.56/6 (9.3%) |
| 2025-11-23 07:41:28,407 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.4%) |
| 2025-11-23 07:42:04,220 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.6%) |
| 2025-11-23 07:43:25,561 - __main__ - INFO - 📊 Evaluation 5: BLEU = 0.1175 |
| 2025-11-23 07:43:25,888 - __main__ - INFO - 📊 Evaluation at step 8000 |
| 2025-11-23 07:43:31,017 - __main__ - INFO - |
| 📤 Pushing checkpoint 8000 to HuggingFace Hub... |
| 2025-11-23 07:43:31,018 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 07:43:39,393 - __main__ - INFO - ✅ Checkpoint 8000 pushed successfully! |
| 2025-11-23 07:43:39,395 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-8000 |
| 2025-11-23 07:43:59,385 - __main__ - INFO - 📈 Epoch progress: 0.58/6 (9.7%) |
| 2025-11-23 07:44:19,912 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.8%) |
| 2025-11-23 07:44:40,386 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.9%) |
| 2025-11-23 07:45:00,848 - __main__ - INFO - 📈 Epoch progress: 0.60/6 (10.0%) |
| 2025-11-23 07:45:20,858 - __main__ - INFO - 📈 Epoch progress: 0.61/6 (10.2%) |
| 2025-11-23 07:45:41,250 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.3%) |
| 2025-11-23 07:46:01,676 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.4%) |
| 2025-11-23 07:46:21,983 - __main__ - INFO - 📈 Epoch progress: 0.63/6 (10.5%) |
| 2025-11-23 07:46:42,221 - __main__ - INFO - 📈 Epoch progress: 0.64/6 (10.6%) |
| 2025-11-23 07:47:02,695 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.8%) |
| 2025-11-23 07:47:22,928 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.9%) |
| 2025-11-23 07:47:37,482 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:47:47,339 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:47:47,340 - __main__ - INFO - 📈 Epoch progress: 0.66/6 (11.0%) |
| 2025-11-23 07:47:47,677 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:47:57,719 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 07:48:07,924 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.1%) |
| 2025-11-23 07:48:29,384 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.2%) |
| 2025-11-23 07:48:49,792 - __main__ - INFO - 📈 Epoch progress: 0.68/6 (11.3%) |
| 2025-11-23 07:49:10,658 - __main__ - INFO - 📈 Epoch progress: 0.69/6 (11.5%) |
| 2025-11-23 07:49:31,428 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.6%) |
| 2025-11-23 07:49:52,589 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.7%) |
| 2025-11-23 07:50:13,540 - __main__ - INFO - 📈 Epoch progress: 0.71/6 (11.8%) |
| 2025-11-23 07:50:33,579 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (11.9%) |
| 2025-11-23 07:51:21,978 - __main__ - INFO - 📊 Evaluation 6: BLEU = 0.1201 |
| 2025-11-23 07:51:22,310 - __main__ - INFO - 📊 Evaluation at step 10000 |
| 2025-11-23 07:51:26,467 - __main__ - INFO - |
| 📤 Pushing checkpoint 10000 to HuggingFace Hub... |
| 2025-11-23 07:51:26,468 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 07:51:36,822 - __main__ - INFO - ✅ Checkpoint 10000 pushed successfully! |
| 2025-11-23 07:51:36,824 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-10000 |
| 2025-11-23 07:51:57,838 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (12.1%) |
| 2025-11-23 07:52:19,175 - __main__ - INFO - 📈 Epoch progress: 0.73/6 (12.2%) |
| 2025-11-23 07:52:39,834 - __main__ - INFO - 📈 Epoch progress: 0.74/6 (12.3%) |
| 2025-11-23 07:53:01,284 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.4%) |
| 2025-11-23 07:53:21,589 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.5%) |
| 2025-11-23 07:53:42,477 - __main__ - INFO - 📈 Epoch progress: 0.76/6 (12.7%) |
| 2025-11-23 07:54:04,122 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.8%) |
| 2025-11-23 07:54:25,232 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.9%) |
| 2025-11-23 07:54:46,194 - __main__ - INFO - 📈 Epoch progress: 0.78/6 (13.0%) |
| 2025-11-23 07:55:07,609 - __main__ - INFO - 📈 Epoch progress: 0.79/6 (13.1%) |
| 2025-11-23 07:55:28,579 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.3%) |
| 2025-11-23 07:55:49,099 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.4%) |
| 2025-11-23 07:56:10,055 - __main__ - INFO - 📈 Epoch progress: 0.81/6 (13.5%) |
| 2025-11-23 07:56:31,036 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.6%) |
| 2025-11-23 07:56:51,675 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.7%) |
| 2025-11-23 07:57:12,699 - __main__ - INFO - 📈 Epoch progress: 0.83/6 (13.9%) |
| 2025-11-23 07:57:33,405 - __main__ - INFO - 📈 Epoch progress: 0.84/6 (14.0%) |
| 2025-11-23 07:57:54,084 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.1%) |
| 2025-11-23 07:58:15,527 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.2%) |
| 2025-11-23 07:58:36,159 - __main__ - INFO - 📈 Epoch progress: 0.86/6 (14.3%) |
| 2025-11-23 07:59:31,105 - __main__ - INFO - 📊 Evaluation 7: BLEU = 0.1172 |
| 2025-11-23 07:59:31,811 - __main__ - INFO - 📊 Evaluation at step 12000 |
| 2025-11-23 07:59:36,349 - __main__ - INFO - |
| 📤 Pushing checkpoint 12000 to HuggingFace Hub... |
| 2025-11-23 07:59:36,350 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 07:59:43,697 - __main__ - INFO - ✅ Checkpoint 12000 pushed successfully! |
| 2025-11-23 07:59:43,698 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-12000 |
| 2025-11-23 07:59:55,944 - __main__ - INFO - 📊 Evaluation 16: BLEU = 0.1289 |
| 2025-11-23 07:59:56,301 - __main__ - INFO - 📊 Evaluation at step 28000 |
| 2025-11-23 07:59:56,472 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-23 07:59:56,528 - __main__ - INFO - Computing predictions for COMET (1000 samples, batch_size=32)... |
| 2025-11-23 07:59:56,529 - __main__ - INFO - Batch 1/32... |
| 2025-11-23 08:00:04,186 - __main__ - INFO - Batch 5/32... |
| 2025-11-23 08:00:06,674 - __main__ - INFO - Batch 10/32... |
| 2025-11-23 08:00:08,376 - __main__ - INFO - Batch 15/32... |
| 2025-11-23 08:00:08,387 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.5%) |
| 2025-11-23 08:00:10,797 - __main__ - INFO - Batch 20/32... |
| 2025-11-23 08:00:13,377 - __main__ - INFO - Batch 25/32... |
| 2025-11-23 08:00:15,932 - __main__ - INFO - Batch 30/32... |
| 2025-11-23 08:00:17,281 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-23 08:00:17,287 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-23 08:00:17,296 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-23 08:00:17,297 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-23 08:00:17,299 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-23 08:00:25,701 - __main__ - INFO - ✅ COMET-QE: 0.1763 ± 0.2628 |
| 2025-11-23 08:00:25,716 - __main__ - INFO - |
| ✅ FINAL Test Results: |
| 2025-11-23 08:00:25,718 - __main__ - INFO - BLEU: 0.1289 |
| 2025-11-23 08:00:25,718 - __main__ - INFO - chrF: 32.25 |
| 2025-11-23 08:00:25,719 - __main__ - INFO - COMET-QE: 0.1763 |
| 2025-11-23 08:00:25,720 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 08:00:25,721 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 08:00:25,721 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-23 08:00:25,722 - __main__ - INFO - ====================================================================== |
| 2025-11-23 08:00:25,723 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-23 08:00:25,724 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-23 08:00:27,792 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-23 08:00:27,871 - __main__ - INFO - ✅ Domain chrF: 1.18 |
| 2025-11-23 08:00:27,872 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-23 08:00:27,876 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-23 08:00:27,885 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-23 08:00:27,886 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-23 08:00:27,888 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-23 08:00:30,990 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 |
| 2025-11-23 08:00:30,991 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-23 08:00:30,992 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-23 08:00:30,992 - __main__ - INFO - chrF: 1.18 |
| 2025-11-23 08:00:30,992 - __main__ - INFO - COMET-QE: 0.0000 |
| 2025-11-23 08:00:30,993 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 08:00:31,015 - __main__ - INFO - 💾 Final results saved to models/finetuned-sw-en-phase2/test_results.json |
| 2025-11-23 08:00:31,022 - __main__ - INFO - ✅ Verified test_results.json with 14 metrics |
| 2025-11-23 08:00:31,023 - __main__ - INFO - |
| 📊 Logging all metrics to TensorBoard... |
| 2025-11-23 08:00:31,177 - __main__ - INFO - ✅ All metrics logged to TensorBoard |
| 2025-11-23 08:00:31,179 - __main__ - INFO - Baseline metrics: 13 |
| 2025-11-23 08:00:31,179 - __main__ - INFO - Final test metrics: 10 |
| 2025-11-23 08:00:31,180 - __main__ - INFO - Final domain metrics: 4 |
| 2025-11-23 08:00:31,181 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 08:00:31,181 - __main__ - INFO - 📊 BASELINE vs FINE-TUNED Comparison |
| 2025-11-23 08:00:31,182 - __main__ - INFO - ====================================================================== |
| 2025-11-23 08:00:31,183 - __main__ - INFO - |
| 📈 Test Set Metrics: |
| 2025-11-23 08:00:31,183 - __main__ - INFO - Metric Baseline Fine-tuned Improvement |
| 2025-11-23 08:00:31,184 - __main__ - INFO - ------------------------------------------------------------ |
| 2025-11-23 08:00:31,185 - __main__ - INFO - eval_bleu 0.0000 0.1289 N/A |
| 2025-11-23 08:00:31,186 - __main__ - INFO - eval_chrf 0.0000 32.2491 N/A |
| 2025-11-23 08:00:31,186 - __main__ - INFO - eval_comet 0.0000 0.1763 N/A |
| 2025-11-23 08:00:31,187 - __main__ - INFO - |
| 📈 Domain Metrics (Real Call Transcriptions): |
| 2025-11-23 08:00:31,188 - __main__ - INFO - Metric Baseline Fine-tuned Improvement |
| 2025-11-23 08:00:31,189 - __main__ - INFO - ------------------------------------------------------------ |
| 2025-11-23 08:00:31,190 - __main__ - INFO - domain_bleu 0.0000 0.0000 +1417.7% |
| 2025-11-23 08:00:31,191 - __main__ - INFO - domain_chrf 1.2250 1.1786 -3.8% |
| 2025-11-23 08:00:31,192 - __main__ - INFO - domain_comet 0.0001 0.0000 -26.2% |
| 2025-11-23 08:00:31,192 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 08:00:32,089 - __main__ - INFO - ✅ Logged 14 final metrics to MLflow |
| 2025-11-23 08:00:32,090 - __main__ - INFO - Visit MLflow: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 08:00:32,091 - __main__ - INFO - 📊 Captured 576 training log entries |
| 2025-11-23 08:00:32,126 - __main__ - INFO - 💾 Saved comprehensive metrics to models/finetuned-sw-en-phase2/training_metrics.json |
| 2025-11-23 08:00:32,143 - __main__ - INFO - 💾 Saved training config to models/finetuned-sw-en-phase2/training_config_used.json |
| 2025-11-23 08:00:32,178 - __main__ - INFO - 💾 Copied training.log to output directory |
| 2025-11-23 08:00:32,179 - __main__ - INFO - |
| 🤖 HUGGINGFACE HUB |
| 2025-11-23 08:00:32,180 - __main__ - INFO - ℹ️ Checkpoints were pushed during training (push_strategy='checkpoint') |
| 2025-11-23 08:00:32,181 - __main__ - INFO - 📤 Pushing final model to hub... |
| 2025-11-23 08:00:32,181 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 08:00:32,182 - __main__ - INFO - 📤 Uploading model to HuggingFace Hub... |
| 2025-11-23 08:00:32,182 - __main__ - INFO - ====================================================================== |
| 2025-11-23 08:00:32,183 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:00:32,190 - __main__ - INFO - ✅ Model card generated |
| 2025-11-23 08:00:32,333 - huggingface_hub._login - WARNING - Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you've just configured. |
| 2025-11-23 08:00:32,334 - __main__ - INFO - Uploading files... |
| 2025-11-23 08:00:32,369 - __main__ - INFO - ✅ Repository created/verified: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:00:32,374 - __main__ - INFO - 📊 Including TensorBoard logs (4 files) |
| 2025-11-23 08:00:32,376 - __main__ - INFO - 📁 Uploading 21 items from models/finetuned-sw-en-phase2 |
| 2025-11-23 08:00:32,377 - __main__ - INFO - 📄 README.md |
| 2025-11-23 08:00:32,379 - __main__ - INFO - 📄 training.log |
| 2025-11-23 08:00:32,380 - __main__ - INFO - 📄 training_config_used.json |
| 2025-11-23 08:00:32,381 - __main__ - INFO - 📄 training_metrics.json |
| 2025-11-23 08:00:32,382 - __main__ - INFO - 📄 test_results.json |
| 2025-11-23 08:00:32,383 - __main__ - INFO - 📂 checkpoint-12000 |
| 2025-11-23 08:00:32,385 - __main__ - INFO - 📂 checkpoint-10000 |
| 2025-11-23 08:00:32,386 - __main__ - INFO - 📂 checkpoint-8000 |
| 2025-11-23 08:00:32,387 - __main__ - INFO - 📄 train_results.json |
| 2025-11-23 08:00:32,388 - __main__ - INFO - 📄 translation_config.json |
| 2025-11-23 08:00:32,390 - __main__ - INFO - 📄 training_args.bin |
| 2025-11-23 08:00:32,391 - __main__ - INFO - 📄 target.spm |
| 2025-11-23 08:00:32,392 - __main__ - INFO - 📄 source.spm |
| 2025-11-23 08:00:32,393 - __main__ - INFO - 📄 vocab.json |
| 2025-11-23 08:00:32,394 - __main__ - INFO - 📄 special_tokens_map.json |
| 2025-11-23 08:00:32,395 - __main__ - INFO - 📄 tokenizer_config.json |
| 2025-11-23 08:00:32,397 - __main__ - INFO - 📄 model.safetensors |
| 2025-11-23 08:00:32,398 - __main__ - INFO - 📄 generation_config.json |
| 2025-11-23 08:00:32,398 - __main__ - INFO - 📄 config.json |
| 2025-11-23 08:00:32,415 - __main__ - INFO - 📄 baseline_results.json |
| 2025-11-23 08:00:32,416 - __main__ - INFO - 📂 logs |
| 2025-11-23 08:00:34,114 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.6%) |
| 2025-11-23 08:00:50,055 - __main__ - INFO - |
| ✅ Model uploaded successfully! |
| 2025-11-23 08:00:50,057 - __main__ - INFO - 🔗 View model at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:00:50,059 - __main__ - INFO - 📊 TensorBoard logs at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/logs |
| 2025-11-23 08:00:50,060 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 08:00:50,061 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 08:00:50,062 - __main__ - INFO - ✅ Pipeline completed successfully! |
| 2025-11-23 08:00:50,062 - __main__ - INFO - ====================================================================== |
| 2025-11-23 08:00:50,063 - __main__ - INFO - |
| 📋 PIPELINE SUMMARY |
| 2025-11-23 08:00:50,064 - __main__ - INFO - ====================================================================== |
| 2025-11-23 08:00:50,065 - __main__ - INFO - |
| ✓ Baseline Evaluation |
| 2025-11-23 08:00:50,066 - __main__ - INFO - - Test set: 210087 samples |
| 2025-11-23 08:00:50,067 - __main__ - INFO - - Domain data: 5 samples |
| 2025-11-23 08:00:50,068 - __main__ - INFO - |
| ✓ Training |
| 2025-11-23 08:00:50,069 - __main__ - INFO - - Epochs: 6 |
| 2025-11-23 08:00:50,069 - __main__ - INFO - - Train samples: 3,571,465 |
| 2025-11-23 08:00:50,070 - __main__ - INFO - - Validation samples: 420,173 |
| 2025-11-23 08:00:50,071 - __main__ - INFO - |
| ✓ Final Evaluation |
| 2025-11-23 08:00:50,072 - __main__ - INFO - - Test set BLEU: 0.1289 |
| 2025-11-23 08:00:50,073 - __main__ - INFO - - Domain BLEU: 0.0000 |
| 2025-11-23 08:00:50,073 - __main__ - INFO - |
| 📁 Output Files |
| 2025-11-23 08:00:50,081 - __main__ - INFO - - Models: /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-23 08:00:50,083 - __main__ - INFO - - Baseline results: models/finetuned-sw-en-phase2/baseline_results.json |
| 2025-11-23 08:00:50,083 - __main__ - INFO - - Test results: models/finetuned-sw-en-phase2/test_results.json |
| 2025-11-23 08:00:50,084 - __main__ - INFO - - Training logs: /workspace/franklin/openchs_rnd/tasks/translation/swahili/training.log |
| 2025-11-23 08:00:50,085 - __main__ - INFO - |
| 📈 MLflow Tracking |
| 2025-11-23 08:00:50,086 - __main__ - INFO - - URL: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 08:00:50,087 - __main__ - INFO - - Run ID: e2e357df5cbe463abba0139445600679 |
| 2025-11-23 08:00:50,088 - __main__ - INFO - |
| 📊 TensorBoard Logs |
| 2025-11-23 08:00:50,089 - __main__ - INFO - - View with: tensorboard --logdir /workspace/franklin/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs |
| 2025-11-23 08:00:50,089 - __main__ - INFO - - All metrics logged: baseline, test, domain, and COMET scores |
| 2025-11-23 08:00:50,090 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 08:00:50,091 - __main__ - INFO - 📈 MLflow Tracking: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 08:00:50,092 - __main__ - INFO - |
| Status codes: |
| 2025-11-23 08:00:50,093 - __main__ - INFO - - 0.5 = Baseline evaluation |
| 2025-11-23 08:00:50,093 - __main__ - INFO - - 1.0 = Training started |
| 2025-11-23 08:00:50,094 - __main__ - INFO - - 2.0 = Training finished |
| 2025-11-23 08:00:50,096 - __main__ - INFO - - 3.0 = Final evaluation |
| 2025-11-23 08:00:50,849 - __main__ - INFO - |
| ✅ MLflow: Pipeline completed - status_code=4.0 |
| 2025-11-23 08:00:51,659 - __main__ - INFO - ✅ MLflow run ended: e2e357df5cbe463abba0139445600679 |
| 2025-11-23 08:00:51,660 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 08:00:55,562 - __main__ - INFO - 📈 Epoch progress: 0.88/6 (14.7%) |
| 2025-11-23 08:01:15,956 - __main__ - INFO - 📈 Epoch progress: 0.89/6 (14.8%) |
| 2025-11-23 08:01:36,402 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (14.9%) |
| 2025-11-23 08:01:58,072 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:02:07,990 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:02:07,991 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (15.1%) |
| 2025-11-23 08:02:08,717 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:02:18,960 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:02:29,571 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:02:29,572 - __main__ - INFO - 📈 Epoch progress: 0.91/6 (15.2%) |
| 2025-11-23 08:02:29,906 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:02:39,798 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '<!DOCTYPE html> |
| <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]--> |
| <!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]--> |
| <!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]--> |
| <head> |
| <title>Cloudflare Tunnel error | food-accepts-blocked-refinance.trycloudflare.com | Cloudflare</title> |
| <meta charset="UTF-8" /> |
| <meta http-equiv="Content-Type" content="text/html |
| <meta http-equiv="X-UA-Compatible" content="IE=Edge" /> |
| <meta name="robots" content="noindex, nofollow" /> |
| <meta name="viewport" content="width=device-width,initial-scale=1" /> |
| <link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" /> |
|
|
|
|
| <script> |
| (function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.stringify){var e=function(a){var c=document.getElementById("error-feedback-survey"),d=document.getElementById("error-feedback-success"),b=new XMLHttpRequest |
| b.send(JSON.stringify(a)) |
| </script> |
|
|
| <script defer src="https://performance.radar.cloudflare.com/beacon.js"></script> |
| </head> |
| <body> |
| <div id="cf-wrapper"> |
| <div class="cf-alert cf-alert-error cf-cookie-error hidden" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div> |
| <div id="cf-error-details" class="p-0"> |
| <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased"> |
| <h1 class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight"> |
| <span data-translate="error">Error</span> |
| <span>1033</span> |
| </h1> |
| <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">Ray ID: 9a2f2546a63da2c4 &bull |
| <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">2025-11-23 08:02:39 UTC</span> |
| <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">Cloudflare Tunnel error</h2> |
| </header> |
|
|
| <section class="w-240 lg:w-full mx-auto mb-8 lg:px-8"> |
| <div id="what-happened-section" class="w-1/2 md:w-full"> |
| <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_happened">What happened?</h2> |
| <p>You've requested a page on a website (food-accepts-blocked-refinance.trycloudflare.com) that is on the <a href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a> network. The host (food-accepts-blocked-refinance.trycloudflare.com) is configured as a Cloudflare Tunnel, and Cloudflare is currently unable to resolve it. |
| |
| </div> |
|
|
| |
| <div id="resolution-copy-section" class="w-1/2 mt-6 text-15 leading-normal"> |
| <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_can_i_do">What can I do?</h2> |
| <p><strong>If you are a visitor of this website:</strong><br />Please try again in a few minutes.</p><p><strong>If you are the owner of this website:</strong><br />Ensure that cloudflared is running and can reach the network. You may wish to enable <a rel="noopener noreferrer" href="https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/routing-to-tunnel/lb/">load balancing</a> for your tunnel.</p> |
| </div> |
| |
| </section> |
|
|
| <div class="feedback-hidden py-8 text-center" id="error-feedback"> |
| <div id="error-feedback-survey" class="footer-line-wrapper"> |
| Was this page helpful? |
| <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-yes" type="button">Yes</button> |
| <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-no" type="button">No</button> |
| </div> |
| <div class="feedback-success feedback-hidden" id="error-feedback-success"> |
| Thank you for your feedback! |
| </div> |
| </div> |
|
|
|
|
| <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300"> |
| <p class="text-13"> |
| <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">9a2f2546a63da2c4</strong></span> |
| <span class="cf-footer-separator sm:hidden">&bull |
| <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1"> |
| Your IP: |
| <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button> |
| <span class="hidden" id="cf-footer-ip">69.30.85.10</span> |
| <span class="cf-footer-separator sm:hidden">&bull |
| </span> |
| <span class="cf-footer-item sm:block sm:mb-1"><span>Performance & |
| |
| </p> |
| <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal") |
| </div><!-- /.error-footer --> |
|
|
|
|
| </div><!-- /#cf-error-details --> |
| </div><!-- /#cf-wrapper --> |
|
|
| <script> |
| window._cf_translation = {} |
| |
| |
| </script> |
|
|
| </body> |
| </html> |
| ' |
| 2025-11-23 08:02:49,395 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '<!DOCTYPE html> |
| <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]--> |
| <!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]--> |
| <!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]--> |
| <head> |
| <title>Cloudflare Tunnel error | food-accepts-blocked-refinance.trycloudflare.com | Cloudflare</title> |
| <meta charset="UTF-8" /> |
| <meta http-equiv="Content-Type" content="text/html |
| <meta http-equiv="X-UA-Compatible" content="IE=Edge" /> |
| <meta name="robots" content="noindex, nofollow" /> |
| <meta name="viewport" content="width=device-width,initial-scale=1" /> |
| <link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" /> |
|
|
|
|
| <script> |
| (function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.stringify){var e=function(a){var c=document.getElementById("error-feedback-survey"),d=document.getElementById("error-feedback-success"),b=new XMLHttpRequest |
| b.send(JSON.stringify(a)) |
| </script> |
|
|
| <script defer src="https://performance.radar.cloudflare.com/beacon.js"></script> |
| </head> |
| <body> |
| <div id="cf-wrapper"> |
| <div class="cf-alert cf-alert-error cf-cookie-error hidden" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div> |
| <div id="cf-error-details" class="p-0"> |
| <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased"> |
| <h1 class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight"> |
| <span data-translate="error">Error</span> |
| <span>1033</span> |
| </h1> |
| <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">Ray ID: 9a2f2582a0f5a2c4 &bull |
| <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">2025-11-23 08:02:49 UTC</span> |
| <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">Cloudflare Tunnel error</h2> |
| </header> |
|
|
| <section class="w-240 lg:w-full mx-auto mb-8 lg:px-8"> |
| <div id="what-happened-section" class="w-1/2 md:w-full"> |
| <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_happened">What happened?</h2> |
| <p>You've requested a page on a website (food-accepts-blocked-refinance.trycloudflare.com) that is on the <a href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a> network. The host (food-accepts-blocked-refinance.trycloudflare.com) is configured as a Cloudflare Tunnel, and Cloudflare is currently unable to resolve it. |
| |
| </div> |
|
|
| |
| <div id="resolution-copy-section" class="w-1/2 mt-6 text-15 leading-normal"> |
| <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_can_i_do">What can I do?</h2> |
| <p><strong>If you are a visitor of this website:</strong><br />Please try again in a few minutes.</p><p><strong>If you are the owner of this website:</strong><br />Ensure that cloudflared is running and can reach the network. You may wish to enable <a rel="noopener noreferrer" href="https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/routing-to-tunnel/lb/">load balancing</a> for your tunnel.</p> |
| </div> |
| |
| </section> |
|
|
| <div class="feedback-hidden py-8 text-center" id="error-feedback"> |
| <div id="error-feedback-survey" class="footer-line-wrapper"> |
| Was this page helpful? |
| <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-yes" type="button">Yes</button> |
| <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-no" type="button">No</button> |
| </div> |
| <div class="feedback-success feedback-hidden" id="error-feedback-success"> |
| Thank you for your feedback! |
| </div> |
| </div> |
|
|
|
|
| <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300"> |
| <p class="text-13"> |
| <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">9a2f2582a0f5a2c4</strong></span> |
| <span class="cf-footer-separator sm:hidden">&bull |
| <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1"> |
| Your IP: |
| <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button> |
| <span class="hidden" id="cf-footer-ip">69.30.85.10</span> |
| <span class="cf-footer-separator sm:hidden">&bull |
| </span> |
| <span class="cf-footer-item sm:block sm:mb-1"><span>Performance & |
| |
| </p> |
| <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal") |
| </div><!-- /.error-footer --> |
|
|
|
|
| </div><!-- /#cf-error-details --> |
| </div><!-- /#cf-wrapper --> |
|
|
| <script> |
| window._cf_translation = {} |
| |
| |
| </script> |
|
|
| </body> |
| </html> |
| ' |
| 2025-11-23 08:02:49,399 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.3%) |
| 2025-11-23 08:02:49,477 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '<!DOCTYPE html> |
| <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]--> |
| <!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]--> |
| <!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]--> |
| <head> |
| <title>Cloudflare Tunnel error | food-accepts-blocked-refinance.trycloudflare.com | Cloudflare</title> |
| <meta charset="UTF-8" /> |
| <meta http-equiv="Content-Type" content="text/html |
| <meta http-equiv="X-UA-Compatible" content="IE=Edge" /> |
| <meta name="robots" content="noindex, nofollow" /> |
| <meta name="viewport" content="width=device-width,initial-scale=1" /> |
| <link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" /> |
|
|
|
|
| <script> |
| (function(){if(document.addEventListener&&window.XMLHttpRequest&&JSON&&JSON.stringify){var e=function(a){var c=document.getElementById("error-feedback-survey"),d=document.getElementById("error-feedback-success"),b=new XMLHttpRequest |
| b.send(JSON.stringify(a)) |
| </script> |
|
|
| <script defer src="https://performance.radar.cloudflare.com/beacon.js"></script> |
| </head> |
| <body> |
| <div id="cf-wrapper"> |
| <div class="cf-alert cf-alert-error cf-cookie-error hidden" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div> |
| <div id="cf-error-details" class="p-0"> |
| <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-15 antialiased"> |
| <h1 class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight"> |
| <span data-translate="error">Error</span> |
| <span>1033</span> |
| </h1> |
| <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">Ray ID: 9a2f25832124a2c4 &bull |
| <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">2025-11-23 08:02:49 UTC</span> |
| <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">Cloudflare Tunnel error</h2> |
| </header> |
|
|
| <section class="w-240 lg:w-full mx-auto mb-8 lg:px-8"> |
| <div id="what-happened-section" class="w-1/2 md:w-full"> |
| <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_happened">What happened?</h2> |
| <p>You've requested a page on a website (food-accepts-blocked-refinance.trycloudflare.com) that is on the <a href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a> network. The host (food-accepts-blocked-refinance.trycloudflare.com) is configured as a Cloudflare Tunnel, and Cloudflare is currently unable to resolve it. |
| |
| </div> |
|
|
| |
| <div id="resolution-copy-section" class="w-1/2 mt-6 text-15 leading-normal"> |
| <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiased" data-translate="what_can_i_do">What can I do?</h2> |
| <p><strong>If you are a visitor of this website:</strong><br />Please try again in a few minutes.</p><p><strong>If you are the owner of this website:</strong><br />Ensure that cloudflared is running and can reach the network. You may wish to enable <a rel="noopener noreferrer" href="https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/routing-to-tunnel/lb/">load balancing</a> for your tunnel.</p> |
| </div> |
| |
| </section> |
|
|
| <div class="feedback-hidden py-8 text-center" id="error-feedback"> |
| <div id="error-feedback-survey" class="footer-line-wrapper"> |
| Was this page helpful? |
| <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-yes" type="button">Yes</button> |
| <button class="border border-solid bg-white cf-button cursor-pointer ml-4 px-4 py-2 rounded" id="feedback-button-no" type="button">No</button> |
| </div> |
| <div class="feedback-success feedback-hidden" id="error-feedback-success"> |
| Thank you for your feedback! |
| </div> |
| </div> |
|
|
|
|
| <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300"> |
| <p class="text-13"> |
| <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">9a2f25832124a2c4</strong></span> |
| <span class="cf-footer-separator sm:hidden">&bull |
| <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1"> |
| Your IP: |
| <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button> |
| <span class="hidden" id="cf-footer-ip">69.30.85.10</span> |
| <span class="cf-footer-separator sm:hidden">&bull |
| </span> |
| <span class="cf-footer-item sm:block sm:mb-1"><span>Performance & |
| |
| </p> |
| <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal") |
| </div><!-- /.error-footer --> |
|
|
|
|
| </div><!-- /#cf-error-details --> |
| </div><!-- /#cf-wrapper --> |
|
|
| <script> |
| window._cf_translation = {} |
| |
| |
| </script> |
|
|
| </body> |
| </html> |
| ' |
| 2025-11-23 08:03:10,086 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.4%) |
| 2025-11-23 08:03:30,868 - __main__ - INFO - 📈 Epoch progress: 0.93/6 (15.5%) |
| 2025-11-23 08:03:51,127 - __main__ - INFO - 📈 Epoch progress: 0.94/6 (15.6%) |
| 2025-11-23 08:04:11,310 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.8%) |
| 2025-11-23 08:04:31,944 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.9%) |
| 2025-11-23 08:04:52,229 - __main__ - INFO - 📈 Epoch progress: 0.96/6 (16.0%) |
| 2025-11-23 08:05:12,433 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.1%) |
| 2025-11-23 08:05:32,947 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.2%) |
| 2025-11-23 08:05:53,251 - __main__ - INFO - 📈 Epoch progress: 0.98/6 (16.4%) |
| 2025-11-23 08:06:13,503 - __main__ - INFO - 📈 Epoch progress: 0.99/6 (16.5%) |
| 2025-11-23 08:06:33,647 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.6%) |
| 2025-11-23 08:06:56,076 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.7%) |
| 2025-11-23 08:07:42,358 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:07:42,361 - __main__ - INFO - 📊 Evaluation 8: BLEU = 0.1178 |
| 2025-11-23 08:07:42,660 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:07:42,661 - __main__ - INFO - 📊 Evaluation at step 14000 |
| 2025-11-23 08:07:50,686 - __main__ - INFO - |
| 📤 Pushing checkpoint 14000 to HuggingFace Hub... |
| 2025-11-23 08:07:50,687 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:08:00,297 - __main__ - INFO - ✅ Checkpoint 14000 pushed successfully! |
| 2025-11-23 08:08:00,298 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-14000 |
| 2025-11-23 08:08:20,043 - __main__ - INFO - 📈 Epoch progress: 1.01/6 (16.8%) |
| 2025-11-23 08:08:40,333 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.0%) |
| 2025-11-23 08:09:00,458 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.1%) |
| 2025-11-23 08:09:20,954 - __main__ - INFO - 📈 Epoch progress: 1.03/6 (17.2%) |
| 2025-11-23 08:09:41,182 - __main__ - INFO - 📈 Epoch progress: 1.04/6 (17.3%) |
| 2025-11-23 08:10:01,521 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.4%) |
| 2025-11-23 08:10:22,100 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.6%) |
| 2025-11-23 08:10:42,424 - __main__ - INFO - 📈 Epoch progress: 1.06/6 (17.7%) |
| 2025-11-23 08:11:03,105 - __main__ - INFO - 📈 Epoch progress: 1.07/6 (17.8%) |
| 2025-11-23 08:11:23,389 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (17.9%) |
| 2025-11-23 08:11:43,462 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (18.0%) |
| 2025-11-23 08:12:03,440 - __main__ - INFO - 📈 Epoch progress: 1.09/6 (18.2%) |
| 2025-11-23 08:12:23,799 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.3%) |
| 2025-11-23 08:12:35,540 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:12:45,484 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:12:45,484 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.4%) |
| 2025-11-23 08:12:45,795 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:13:06,414 - __main__ - INFO - 📈 Epoch progress: 1.11/6 (18.5%) |
| 2025-11-23 08:13:27,011 - __main__ - INFO - 📈 Epoch progress: 1.12/6 (18.6%) |
| 2025-11-23 08:13:47,468 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.8%) |
| 2025-11-23 08:14:07,883 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.9%) |
| 2025-11-23 08:14:28,371 - __main__ - INFO - 📈 Epoch progress: 1.14/6 (19.0%) |
| 2025-11-23 08:14:48,689 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.1%) |
| 2025-11-23 08:15:36,568 - __main__ - INFO - 📊 Evaluation 9: BLEU = 0.1183 |
| 2025-11-23 08:15:36,909 - __main__ - INFO - 📊 Evaluation at step 16000 |
| 2025-11-23 08:15:40,691 - __main__ - INFO - |
| 📤 Pushing checkpoint 16000 to HuggingFace Hub... |
| 2025-11-23 08:15:40,692 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:15:48,673 - __main__ - INFO - ✅ Checkpoint 16000 pushed successfully! |
| 2025-11-23 08:15:48,674 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-16000 |
| 2025-11-23 08:16:08,416 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.2%) |
| 2025-11-23 08:16:29,051 - __main__ - INFO - 📈 Epoch progress: 1.16/6 (19.4%) |
| 2025-11-23 08:16:49,428 - __main__ - INFO - 📈 Epoch progress: 1.17/6 (19.5%) |
| 2025-11-23 08:17:10,033 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.6%) |
| 2025-11-23 08:17:30,608 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.7%) |
| 2025-11-23 08:17:51,070 - __main__ - INFO - 📈 Epoch progress: 1.19/6 (19.8%) |
| 2025-11-23 08:18:12,240 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (19.9%) |
| 2025-11-23 08:18:32,443 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (20.1%) |
| 2025-11-23 08:18:52,757 - __main__ - INFO - 📈 Epoch progress: 1.21/6 (20.2%) |
| 2025-11-23 08:19:12,512 - __main__ - INFO - 📈 Epoch progress: 1.22/6 (20.3%) |
| 2025-11-23 08:19:33,339 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.4%) |
| 2025-11-23 08:19:54,208 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.5%) |
| 2025-11-23 08:20:14,632 - __main__ - INFO - 📈 Epoch progress: 1.24/6 (20.7%) |
| 2025-11-23 08:20:35,260 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.8%) |
| 2025-11-23 08:20:55,982 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.9%) |
| 2025-11-23 08:21:16,552 - __main__ - INFO - 📈 Epoch progress: 1.26/6 (21.0%) |
| 2025-11-23 08:21:37,005 - __main__ - INFO - 📈 Epoch progress: 1.27/6 (21.1%) |
| 2025-11-23 08:21:57,316 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.3%) |
| 2025-11-23 08:22:17,535 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.4%) |
| 2025-11-23 08:22:38,230 - __main__ - INFO - 📈 Epoch progress: 1.29/6 (21.5%) |
| 2025-11-23 08:23:24,100 - __main__ - INFO - 📊 Evaluation 10: BLEU = 0.1219 |
| 2025-11-23 08:23:24,479 - __main__ - INFO - 📊 Evaluation at step 18000 |
| 2025-11-23 08:23:28,627 - __main__ - INFO - |
| 📤 Pushing checkpoint 18000 to HuggingFace Hub... |
| 2025-11-23 08:23:28,628 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:23:37,701 - __main__ - INFO - ✅ Checkpoint 18000 pushed successfully! |
| 2025-11-23 08:23:37,716 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-18000 |
| 2025-11-23 08:23:57,837 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.6%) |
| 2025-11-23 08:24:18,077 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.7%) |
| 2025-11-23 08:24:38,762 - __main__ - INFO - 📈 Epoch progress: 1.31/6 (21.9%) |
| 2025-11-23 08:24:58,968 - __main__ - INFO - 📈 Epoch progress: 1.32/6 (22.0%) |
| 2025-11-23 08:25:19,819 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.1%) |
| 2025-11-23 08:25:40,288 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.2%) |
| 2025-11-23 08:26:01,048 - __main__ - INFO - 📈 Epoch progress: 1.34/6 (22.3%) |
| 2025-11-23 08:26:21,607 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.5%) |
| 2025-11-23 08:26:41,975 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.6%) |
| 2025-11-23 08:27:02,121 - __main__ - INFO - 📈 Epoch progress: 1.36/6 (22.7%) |
| 2025-11-23 08:27:22,434 - __main__ - INFO - 📈 Epoch progress: 1.37/6 (22.8%) |
| 2025-11-23 08:27:47,892 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:27:47,893 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (22.9%) |
| 2025-11-23 08:27:48,203 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:28:08,098 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (23.1%) |
| 2025-11-23 08:28:28,471 - __main__ - INFO - 📈 Epoch progress: 1.39/6 (23.2%) |
| 2025-11-23 08:28:49,452 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.3%) |
| 2025-11-23 08:29:09,642 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.4%) |
| 2025-11-23 08:29:29,850 - __main__ - INFO - 📈 Epoch progress: 1.41/6 (23.5%) |
| 2025-11-23 08:29:50,456 - __main__ - INFO - 📈 Epoch progress: 1.42/6 (23.7%) |
| 2025-11-23 08:30:11,200 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.8%) |
| 2025-11-23 08:30:31,277 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.9%) |
| 2025-11-23 08:31:18,892 - __main__ - INFO - 📊 Evaluation 11: BLEU = 0.1225 |
| 2025-11-23 08:31:19,249 - __main__ - INFO - 📊 Evaluation at step 20000 |
| 2025-11-23 08:31:22,663 - __main__ - INFO - |
| 📤 Pushing checkpoint 20000 to HuggingFace Hub... |
| 2025-11-23 08:31:22,665 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:31:29,912 - __main__ - INFO - ✅ Checkpoint 20000 pushed successfully! |
| 2025-11-23 08:31:29,916 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-20000 |
| 2025-11-23 08:31:50,195 - __main__ - INFO - 📈 Epoch progress: 1.44/6 (24.0%) |
| 2025-11-23 08:32:10,350 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.1%) |
| 2025-11-23 08:32:31,382 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.2%) |
| 2025-11-23 08:32:52,241 - __main__ - INFO - 📈 Epoch progress: 1.46/6 (24.4%) |
| 2025-11-23 08:33:12,787 - __main__ - INFO - 📈 Epoch progress: 1.47/6 (24.5%) |
| 2025-11-23 08:33:33,381 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.6%) |
| 2025-11-23 08:33:53,906 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.7%) |
| 2025-11-23 08:34:14,948 - __main__ - INFO - 📈 Epoch progress: 1.49/6 (24.8%) |
| 2025-11-23 08:34:35,256 - __main__ - INFO - 📈 Epoch progress: 1.50/6 (25.0%) |
| 2025-11-23 08:34:56,451 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:34:56,453 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.1%) |
| 2025-11-23 08:34:56,755 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:35:17,468 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.2%) |
| 2025-11-23 08:35:38,056 - __main__ - INFO - 📈 Epoch progress: 1.52/6 (25.3%) |
| 2025-11-23 08:35:58,638 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.4%) |
| 2025-11-23 08:36:19,131 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.6%) |
| 2025-11-23 08:36:39,411 - __main__ - INFO - 📈 Epoch progress: 1.54/6 (25.7%) |
| 2025-11-23 08:37:00,265 - __main__ - INFO - 📈 Epoch progress: 1.55/6 (25.8%) |
| 2025-11-23 08:37:20,391 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (25.9%) |
| 2025-11-23 08:37:40,850 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (26.0%) |
| 2025-11-23 08:38:00,979 - __main__ - INFO - 📈 Epoch progress: 1.57/6 (26.2%) |
| 2025-11-23 08:38:21,565 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.3%) |
| 2025-11-23 08:39:16,978 - __main__ - INFO - 📊 Evaluation 12: BLEU = 0.1133 |
| 2025-11-23 08:39:17,337 - __main__ - INFO - 📊 Evaluation at step 22000 |
| 2025-11-23 08:39:21,067 - __main__ - INFO - |
| 📤 Pushing checkpoint 22000 to HuggingFace Hub... |
| 2025-11-23 08:39:21,068 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:39:29,124 - __main__ - INFO - ✅ Checkpoint 22000 pushed successfully! |
| 2025-11-23 08:39:29,126 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-22000 |
| 2025-11-23 08:39:49,626 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.4%) |
| 2025-11-23 08:40:10,631 - __main__ - INFO - 📈 Epoch progress: 1.59/6 (26.5%) |
| 2025-11-23 08:40:31,193 - __main__ - INFO - 📈 Epoch progress: 1.60/6 (26.6%) |
| 2025-11-23 08:40:51,268 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.8%) |
| 2025-11-23 08:41:12,552 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.9%) |
| 2025-11-23 08:41:32,697 - __main__ - INFO - 📈 Epoch progress: 1.62/6 (27.0%) |
| 2025-11-23 08:41:53,243 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.1%) |
| 2025-11-23 08:42:13,872 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.2%) |
| 2025-11-23 08:42:34,689 - __main__ - INFO - 📈 Epoch progress: 1.64/6 (27.4%) |
| 2025-11-23 08:42:55,351 - __main__ - INFO - 📈 Epoch progress: 1.65/6 (27.5%) |
| 2025-11-23 08:43:15,850 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.6%) |
| 2025-11-23 08:43:36,025 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.7%) |
| 2025-11-23 08:43:56,321 - __main__ - INFO - 📈 Epoch progress: 1.67/6 (27.8%) |
| 2025-11-23 08:44:16,729 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.0%) |
| 2025-11-23 08:44:37,239 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.1%) |
| 2025-11-23 08:44:57,578 - __main__ - INFO - 📈 Epoch progress: 1.69/6 (28.2%) |
| 2025-11-23 08:45:18,212 - __main__ - INFO - 📈 Epoch progress: 1.70/6 (28.3%) |
| 2025-11-23 08:45:39,001 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.4%) |
| 2025-11-23 08:45:59,703 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.5%) |
| 2025-11-23 08:46:20,019 - __main__ - INFO - 📈 Epoch progress: 1.72/6 (28.7%) |
| 2025-11-23 08:47:02,800 - __main__ - INFO - 📊 Evaluation 13: BLEU = 0.1197 |
| 2025-11-23 08:47:03,154 - __main__ - INFO - 📊 Evaluation at step 24000 |
| 2025-11-23 08:47:07,057 - __main__ - INFO - |
| 📤 Pushing checkpoint 24000 to HuggingFace Hub... |
| 2025-11-23 08:47:07,057 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:47:15,669 - __main__ - INFO - ✅ Checkpoint 24000 pushed successfully! |
| 2025-11-23 08:47:15,671 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-24000 |
| 2025-11-23 08:47:35,776 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.8%) |
| 2025-11-23 08:47:55,967 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.9%) |
| 2025-11-23 08:48:16,189 - __main__ - INFO - 📈 Epoch progress: 1.74/6 (29.0%) |
| 2025-11-23 08:48:36,698 - __main__ - INFO - 📈 Epoch progress: 1.75/6 (29.1%) |
| 2025-11-23 08:48:56,756 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.3%) |
| 2025-11-23 08:49:17,605 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.4%) |
| 2025-11-23 08:49:37,883 - __main__ - INFO - 📈 Epoch progress: 1.77/6 (29.5%) |
| 2025-11-23 08:49:58,863 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.6%) |
| 2025-11-23 08:50:20,068 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.7%) |
| 2025-11-23 08:50:40,801 - __main__ - INFO - 📈 Epoch progress: 1.79/6 (29.9%) |
| 2025-11-23 08:51:01,156 - __main__ - INFO - 📈 Epoch progress: 1.80/6 (30.0%) |
| 2025-11-23 08:51:21,244 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.1%) |
| 2025-11-23 08:51:41,374 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.2%) |
| 2025-11-23 08:52:01,723 - __main__ - INFO - 📈 Epoch progress: 1.82/6 (30.3%) |
| 2025-11-23 08:52:22,182 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.5%) |
| 2025-11-23 08:52:42,735 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.6%) |
| 2025-11-23 08:53:02,880 - __main__ - INFO - 📈 Epoch progress: 1.84/6 (30.7%) |
| 2025-11-23 08:53:23,224 - __main__ - INFO - 📈 Epoch progress: 1.85/6 (30.8%) |
| 2025-11-23 08:53:43,760 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (30.9%) |
| 2025-11-23 08:54:03,729 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (31.1%) |
| 2025-11-23 08:54:46,422 - __main__ - INFO - 📊 Evaluation 14: BLEU = 0.1220 |
| 2025-11-23 08:54:57,420 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 08:54:57,422 - __main__ - INFO - 📊 Evaluation at step 26000 |
| 2025-11-23 08:55:01,217 - __main__ - INFO - |
| 📤 Pushing checkpoint 26000 to HuggingFace Hub... |
| 2025-11-23 08:55:01,218 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 08:55:09,583 - __main__ - INFO - ✅ Checkpoint 26000 pushed successfully! |
| 2025-11-23 08:55:09,586 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-26000 |
| 2025-11-23 08:55:30,132 - __main__ - INFO - 📈 Epoch progress: 1.87/6 (31.2%) |
| 2025-11-23 08:55:50,431 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.3%) |
| 2025-11-23 08:56:11,046 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.4%) |
| 2025-11-23 08:56:31,743 - __main__ - INFO - 📈 Epoch progress: 1.89/6 (31.5%) |
| 2025-11-23 08:56:52,426 - __main__ - INFO - 📈 Epoch progress: 1.90/6 (31.7%) |
| 2025-11-23 08:57:12,972 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.8%) |
| 2025-11-23 08:57:33,001 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.9%) |
| 2025-11-23 08:57:53,460 - __main__ - INFO - 📈 Epoch progress: 1.92/6 (32.0%) |
| 2025-11-23 08:58:13,359 - __main__ - INFO - 📈 Epoch progress: 1.93/6 (32.1%) |
| 2025-11-23 08:58:33,491 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.3%) |
| 2025-11-23 08:58:53,787 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.4%) |
| 2025-11-23 08:59:14,134 - __main__ - INFO - 📈 Epoch progress: 1.95/6 (32.5%) |
| 2025-11-23 08:59:34,412 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.6%) |
| 2025-11-23 08:59:54,653 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.7%) |
| 2025-11-23 09:00:14,806 - __main__ - INFO - 📈 Epoch progress: 1.97/6 (32.8%) |
| 2025-11-23 09:00:35,080 - __main__ - INFO - 📈 Epoch progress: 1.98/6 (33.0%) |
| 2025-11-23 09:00:55,829 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.1%) |
| 2025-11-23 09:01:16,290 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.2%) |
| 2025-11-23 09:01:36,929 - __main__ - INFO - 📈 Epoch progress: 2.00/6 (33.3%) |
| 2025-11-23 09:01:59,922 - __main__ - INFO - 📈 Epoch progress: 2.01/6 (33.4%) |
| 2025-11-23 09:02:38,082 - __main__ - INFO - 📊 Evaluation 15: BLEU = 0.1212 |
| 2025-11-23 09:02:38,384 - __main__ - INFO - 📊 Evaluation at step 28000 |
| 2025-11-23 09:02:41,951 - __main__ - INFO - |
| 📤 Pushing checkpoint 28000 to HuggingFace Hub... |
| 2025-11-23 09:02:41,952 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 09:02:49,639 - __main__ - INFO - ✅ Checkpoint 28000 pushed successfully! |
| 2025-11-23 09:02:49,641 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-28000 |
| 2025-11-23 09:03:09,347 - __main__ - INFO - 📈 Epoch progress: 2.01/6 (33.6%) |
| 2025-11-23 09:03:29,574 - __main__ - INFO - 📈 Epoch progress: 2.02/6 (33.7%) |
| 2025-11-23 09:03:50,131 - __main__ - INFO - 📈 Epoch progress: 2.03/6 (33.8%) |
| 2025-11-23 09:04:10,657 - __main__ - INFO - 📈 Epoch progress: 2.04/6 (33.9%) |
| 2025-11-23 09:04:30,935 - __main__ - INFO - 📈 Epoch progress: 2.04/6 (34.0%) |
| 2025-11-23 09:04:51,362 - __main__ - INFO - 📈 Epoch progress: 2.05/6 (34.2%) |
| 2025-11-23 09:05:12,207 - __main__ - INFO - 📈 Epoch progress: 2.06/6 (34.3%) |
| 2025-11-23 09:05:32,445 - __main__ - INFO - 📈 Epoch progress: 2.06/6 (34.4%) |
| 2025-11-23 09:05:52,852 - __main__ - INFO - 📈 Epoch progress: 2.07/6 (34.5%) |
| 2025-11-23 09:06:13,060 - __main__ - INFO - 📈 Epoch progress: 2.08/6 (34.6%) |
| 2025-11-23 09:06:33,286 - __main__ - INFO - 📈 Epoch progress: 2.09/6 (34.8%) |
| 2025-11-23 09:06:53,703 - __main__ - INFO - 📈 Epoch progress: 2.09/6 (34.9%) |
| 2025-11-23 09:07:14,264 - __main__ - INFO - 📈 Epoch progress: 2.10/6 (35.0%) |
| 2025-11-23 09:07:34,690 - __main__ - INFO - 📈 Epoch progress: 2.11/6 (35.1%) |
| 2025-11-23 09:07:55,165 - __main__ - INFO - 📈 Epoch progress: 2.11/6 (35.2%) |
| 2025-11-23 09:08:15,733 - __main__ - INFO - 📈 Epoch progress: 2.12/6 (35.4%) |
| 2025-11-23 09:08:36,049 - __main__ - INFO - 📈 Epoch progress: 2.13/6 (35.5%) |
| 2025-11-23 09:08:56,649 - __main__ - INFO - 📈 Epoch progress: 2.14/6 (35.6%) |
| 2025-11-23 09:09:17,044 - __main__ - INFO - 📈 Epoch progress: 2.14/6 (35.7%) |
| 2025-11-23 09:09:37,555 - __main__ - INFO - 📈 Epoch progress: 2.15/6 (35.8%) |
| 2025-11-23 09:10:16,737 - __main__ - INFO - 📊 Evaluation 16: BLEU = 0.1249 |
| 2025-11-23 09:10:17,064 - __main__ - INFO - 📊 Evaluation at step 30000 |
| 2025-11-23 09:10:22,179 - __main__ - INFO - |
| 📤 Pushing checkpoint 30000 to HuggingFace Hub... |
| 2025-11-23 09:10:22,180 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 09:10:32,265 - __main__ - INFO - ✅ Checkpoint 30000 pushed successfully! |
| 2025-11-23 09:10:32,267 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-30000 |
| 2025-11-23 09:10:52,200 - __main__ - INFO - 📈 Epoch progress: 2.16/6 (36.0%) |
| 2025-11-23 09:11:12,243 - __main__ - INFO - 📈 Epoch progress: 2.16/6 (36.1%) |
| 2025-11-23 09:11:32,817 - __main__ - INFO - 📈 Epoch progress: 2.17/6 (36.2%) |
| 2025-11-23 09:11:52,857 - __main__ - INFO - 📈 Epoch progress: 2.18/6 (36.3%) |
| 2025-11-23 09:12:13,030 - __main__ - INFO - 📈 Epoch progress: 2.19/6 (36.4%) |
| 2025-11-23 09:12:37,455 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:12:37,457 - __main__ - INFO - 📈 Epoch progress: 2.19/6 (36.6%) |
| 2025-11-23 09:12:37,777 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:12:47,576 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:12:57,551 - __main__ - INFO - 📈 Epoch progress: 2.20/6 (36.7%) |
| 2025-11-23 09:13:18,678 - __main__ - INFO - 📈 Epoch progress: 2.21/6 (36.8%) |
| 2025-11-23 09:13:39,512 - __main__ - INFO - 📈 Epoch progress: 2.21/6 (36.9%) |
| 2025-11-23 09:14:00,077 - __main__ - INFO - 📈 Epoch progress: 2.22/6 (37.0%) |
| 2025-11-23 09:14:20,418 - __main__ - INFO - 📈 Epoch progress: 2.23/6 (37.1%) |
| 2025-11-23 09:14:40,797 - __main__ - INFO - 📈 Epoch progress: 2.24/6 (37.3%) |
| 2025-11-23 09:15:01,178 - __main__ - INFO - 📈 Epoch progress: 2.24/6 (37.4%) |
| 2025-11-23 09:15:21,882 - __main__ - INFO - 📈 Epoch progress: 2.25/6 (37.5%) |
| 2025-11-23 09:15:43,229 - __main__ - INFO - 📈 Epoch progress: 2.26/6 (37.6%) |
| 2025-11-23 09:16:03,702 - __main__ - INFO - 📈 Epoch progress: 2.26/6 (37.7%) |
| 2025-11-23 09:16:23,964 - __main__ - INFO - 📈 Epoch progress: 2.27/6 (37.9%) |
| 2025-11-23 09:16:44,557 - __main__ - INFO - 📈 Epoch progress: 2.28/6 (38.0%) |
| 2025-11-23 09:17:05,127 - __main__ - INFO - 📈 Epoch progress: 2.29/6 (38.1%) |
| 2025-11-23 09:17:25,409 - __main__ - INFO - 📈 Epoch progress: 2.29/6 (38.2%) |
| 2025-11-23 09:18:08,617 - __main__ - INFO - 📊 Evaluation 17: BLEU = 0.1247 |
| 2025-11-23 09:18:08,956 - __main__ - INFO - 📊 Evaluation at step 32000 |
| 2025-11-23 09:18:12,719 - __main__ - INFO - |
| 📤 Pushing checkpoint 32000 to HuggingFace Hub... |
| 2025-11-23 09:18:12,720 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 09:18:22,296 - __main__ - INFO - ✅ Checkpoint 32000 pushed successfully! |
| 2025-11-23 09:18:22,298 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-32000 |
| 2025-11-23 09:18:42,074 - __main__ - INFO - 📈 Epoch progress: 2.30/6 (38.3%) |
| 2025-11-23 09:19:02,439 - __main__ - INFO - 📈 Epoch progress: 2.31/6 (38.5%) |
| 2025-11-23 09:19:31,259 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:19:31,262 - __main__ - INFO - 📈 Epoch progress: 2.31/6 (38.6%) |
| 2025-11-23 09:19:31,555 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:19:41,425 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:19:51,215 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:19:51,218 - __main__ - INFO - 📈 Epoch progress: 2.32/6 (38.7%) |
| 2025-11-23 09:19:52,068 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:20:12,055 - __main__ - INFO - 📈 Epoch progress: 2.33/6 (38.8%) |
| 2025-11-23 09:20:32,715 - __main__ - INFO - 📈 Epoch progress: 2.34/6 (38.9%) |
| 2025-11-23 09:20:52,820 - __main__ - INFO - 📈 Epoch progress: 2.34/6 (39.1%) |
| 2025-11-23 09:21:13,225 - __main__ - INFO - 📈 Epoch progress: 2.35/6 (39.2%) |
| 2025-11-23 09:21:33,172 - __main__ - INFO - 📈 Epoch progress: 2.36/6 (39.3%) |
| 2025-11-23 09:21:53,617 - __main__ - INFO - 📈 Epoch progress: 2.37/6 (39.4%) |
| 2025-11-23 09:22:14,037 - __main__ - INFO - 📈 Epoch progress: 2.37/6 (39.5%) |
| 2025-11-23 09:22:34,315 - __main__ - INFO - 📈 Epoch progress: 2.38/6 (39.7%) |
| 2025-11-23 09:22:54,536 - __main__ - INFO - 📈 Epoch progress: 2.39/6 (39.8%) |
| 2025-11-23 09:23:15,047 - __main__ - INFO - 📈 Epoch progress: 2.39/6 (39.9%) |
| 2025-11-23 09:23:35,587 - __main__ - INFO - 📈 Epoch progress: 2.40/6 (40.0%) |
| 2025-11-23 09:23:55,756 - __main__ - INFO - 📈 Epoch progress: 2.41/6 (40.1%) |
| 2025-11-23 09:24:16,224 - __main__ - INFO - 📈 Epoch progress: 2.42/6 (40.3%) |
| 2025-11-23 09:24:36,699 - __main__ - INFO - 📈 Epoch progress: 2.42/6 (40.4%) |
| 2025-11-23 09:25:01,406 - __main__ - INFO - 📈 Epoch progress: 2.43/6 (40.5%) |
| 2025-11-23 09:25:21,453 - __main__ - INFO - 📈 Epoch progress: 2.44/6 (40.6%) |
| 2025-11-23 09:26:03,414 - __main__ - INFO - 📊 Evaluation 18: BLEU = 0.1201 |
| 2025-11-23 09:26:03,728 - __main__ - INFO - 📊 Evaluation at step 34000 |
| 2025-11-23 09:26:07,747 - __main__ - INFO - |
| 📤 Pushing checkpoint 34000 to HuggingFace Hub... |
| 2025-11-23 09:26:07,748 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 09:26:17,915 - __main__ - INFO - ✅ Checkpoint 34000 pushed successfully! |
| 2025-11-23 09:26:17,916 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-34000 |
| 2025-11-23 09:26:38,257 - __main__ - INFO - 📈 Epoch progress: 2.44/6 (40.7%) |
| 2025-11-23 09:26:58,506 - __main__ - INFO - 📈 Epoch progress: 2.45/6 (40.9%) |
| 2025-11-23 09:27:18,774 - __main__ - INFO - 📈 Epoch progress: 2.46/6 (41.0%) |
| 2025-11-23 09:27:39,202 - __main__ - INFO - 📈 Epoch progress: 2.47/6 (41.1%) |
| 2025-11-23 09:27:59,546 - __main__ - INFO - 📈 Epoch progress: 2.47/6 (41.2%) |
| 2025-11-23 09:28:19,784 - __main__ - INFO - 📈 Epoch progress: 2.48/6 (41.3%) |
| 2025-11-23 09:28:40,124 - __main__ - INFO - 📈 Epoch progress: 2.49/6 (41.4%) |
| 2025-11-23 09:29:00,552 - __main__ - INFO - 📈 Epoch progress: 2.49/6 (41.6%) |
| 2025-11-23 09:29:20,903 - __main__ - INFO - 📈 Epoch progress: 2.50/6 (41.7%) |
| 2025-11-23 09:29:41,466 - __main__ - INFO - 📈 Epoch progress: 2.51/6 (41.8%) |
| 2025-11-23 09:30:01,776 - __main__ - INFO - 📈 Epoch progress: 2.52/6 (41.9%) |
| 2025-11-23 09:30:22,006 - __main__ - INFO - 📈 Epoch progress: 2.52/6 (42.0%) |
| 2025-11-23 09:30:42,360 - __main__ - INFO - 📈 Epoch progress: 2.53/6 (42.2%) |
| 2025-11-23 09:31:02,482 - __main__ - INFO - 📈 Epoch progress: 2.54/6 (42.3%) |
| 2025-11-23 09:31:22,826 - __main__ - INFO - 📈 Epoch progress: 2.54/6 (42.4%) |
| 2025-11-23 09:31:43,466 - __main__ - INFO - 📈 Epoch progress: 2.55/6 (42.5%) |
| 2025-11-23 09:32:04,045 - __main__ - INFO - 📈 Epoch progress: 2.56/6 (42.6%) |
| 2025-11-23 09:32:24,371 - __main__ - INFO - 📈 Epoch progress: 2.57/6 (42.8%) |
| 2025-11-23 09:32:44,985 - __main__ - INFO - 📈 Epoch progress: 2.57/6 (42.9%) |
| 2025-11-23 09:33:05,336 - __main__ - INFO - 📈 Epoch progress: 2.58/6 (43.0%) |
| 2025-11-23 09:33:54,033 - __main__ - INFO - 📊 Evaluation 19: BLEU = 0.1199 |
| 2025-11-23 09:33:54,344 - __main__ - INFO - 📊 Evaluation at step 36000 |
| 2025-11-23 09:33:59,171 - __main__ - INFO - |
| 📤 Pushing checkpoint 36000 to HuggingFace Hub... |
| 2025-11-23 09:33:59,173 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 09:34:11,715 - __main__ - INFO - ✅ Checkpoint 36000 pushed successfully! |
| 2025-11-23 09:34:11,716 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-36000 |
| 2025-11-23 09:34:31,557 - __main__ - INFO - 📈 Epoch progress: 2.59/6 (43.1%) |
| 2025-11-23 09:34:52,089 - __main__ - INFO - 📈 Epoch progress: 2.59/6 (43.2%) |
| 2025-11-23 09:35:12,625 - __main__ - INFO - 📈 Epoch progress: 2.60/6 (43.4%) |
| 2025-11-23 09:35:33,297 - __main__ - INFO - 📈 Epoch progress: 2.61/6 (43.5%) |
| 2025-11-23 09:35:53,392 - __main__ - INFO - 📈 Epoch progress: 2.62/6 (43.6%) |
| 2025-11-23 09:36:13,883 - __main__ - INFO - 📈 Epoch progress: 2.62/6 (43.7%) |
| 2025-11-23 09:36:34,079 - __main__ - INFO - 📈 Epoch progress: 2.63/6 (43.8%) |
| 2025-11-23 09:36:54,116 - __main__ - INFO - 📈 Epoch progress: 2.64/6 (44.0%) |
| 2025-11-23 09:37:14,800 - __main__ - INFO - 📈 Epoch progress: 2.64/6 (44.1%) |
| 2025-11-23 09:37:35,136 - __main__ - INFO - 📈 Epoch progress: 2.65/6 (44.2%) |
| 2025-11-23 09:37:55,597 - __main__ - INFO - 📈 Epoch progress: 2.66/6 (44.3%) |
| 2025-11-23 09:38:16,316 - __main__ - INFO - 📈 Epoch progress: 2.67/6 (44.4%) |
| 2025-11-23 09:40:15,894 - __main__ - INFO - 📈 Epoch progress: 2.67/6 (44.6%) |
| 2025-11-23 09:40:36,039 - __main__ - INFO - 📈 Epoch progress: 2.68/6 (44.7%) |
| 2025-11-23 09:40:55,973 - __main__ - INFO - 📈 Epoch progress: 2.69/6 (44.8%) |
| 2025-11-23 09:41:16,576 - __main__ - INFO - 📈 Epoch progress: 2.69/6 (44.9%) |
| 2025-11-23 09:41:37,086 - __main__ - INFO - 📈 Epoch progress: 2.70/6 (45.0%) |
| 2025-11-23 09:41:57,577 - __main__ - INFO - 📈 Epoch progress: 2.71/6 (45.2%) |
| 2025-11-23 09:42:17,849 - __main__ - INFO - 📈 Epoch progress: 2.72/6 (45.3%) |
| 2025-11-23 09:42:38,187 - __main__ - INFO - 📈 Epoch progress: 2.72/6 (45.4%) |
| 2025-11-23 09:43:19,479 - __main__ - INFO - 📊 Evaluation 20: BLEU = 0.1221 |
| 2025-11-23 09:43:19,792 - __main__ - INFO - 📊 Evaluation at step 38000 |
| 2025-11-23 09:43:24,361 - __main__ - INFO - |
| 📤 Pushing checkpoint 38000 to HuggingFace Hub... |
| 2025-11-23 09:43:24,362 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 09:43:34,489 - __main__ - INFO - ✅ Checkpoint 38000 pushed successfully! |
| 2025-11-23 09:43:34,491 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-38000 |
| 2025-11-23 09:43:54,899 - __main__ - INFO - 📈 Epoch progress: 2.73/6 (45.5%) |
| 2025-11-23 09:44:15,186 - __main__ - INFO - 📈 Epoch progress: 2.74/6 (45.6%) |
| 2025-11-23 09:44:35,248 - __main__ - INFO - 📈 Epoch progress: 2.74/6 (45.7%) |
| 2025-11-23 09:44:55,457 - __main__ - INFO - 📈 Epoch progress: 2.75/6 (45.9%) |
| 2025-11-23 09:45:16,526 - __main__ - INFO - 📈 Epoch progress: 2.76/6 (46.0%) |
| 2025-11-23 09:45:36,727 - __main__ - INFO - 📈 Epoch progress: 2.77/6 (46.1%) |
| 2025-11-23 09:45:57,334 - __main__ - INFO - 📈 Epoch progress: 2.77/6 (46.2%) |
| 2025-11-23 09:46:17,813 - __main__ - INFO - 📈 Epoch progress: 2.78/6 (46.3%) |
| 2025-11-23 09:46:38,336 - __main__ - INFO - 📈 Epoch progress: 2.79/6 (46.5%) |
| 2025-11-23 09:46:58,873 - __main__ - INFO - 📈 Epoch progress: 2.80/6 (46.6%) |
| 2025-11-23 09:47:19,339 - __main__ - INFO - 📈 Epoch progress: 2.80/6 (46.7%) |
| 2025-11-23 09:47:39,806 - __main__ - INFO - 📈 Epoch progress: 2.81/6 (46.8%) |
| 2025-11-23 09:48:00,195 - __main__ - INFO - 📈 Epoch progress: 2.82/6 (46.9%) |
| 2025-11-23 09:48:20,524 - __main__ - INFO - 📈 Epoch progress: 2.82/6 (47.1%) |
| 2025-11-23 09:48:40,993 - __main__ - INFO - 📈 Epoch progress: 2.83/6 (47.2%) |
| 2025-11-23 09:49:01,234 - __main__ - INFO - 📈 Epoch progress: 2.84/6 (47.3%) |
| 2025-11-23 09:49:21,981 - __main__ - INFO - 📈 Epoch progress: 2.85/6 (47.4%) |
| 2025-11-23 09:49:42,732 - __main__ - INFO - 📈 Epoch progress: 2.85/6 (47.5%) |
| 2025-11-23 09:50:03,204 - __main__ - INFO - 📈 Epoch progress: 2.86/6 (47.7%) |
| 2025-11-23 09:50:23,702 - __main__ - INFO - 📈 Epoch progress: 2.87/6 (47.8%) |
| 2025-11-23 09:50:59,183 - __main__ - INFO - 📊 Evaluation 21: BLEU = 0.1263 |
| 2025-11-23 09:50:59,500 - __main__ - INFO - 📊 Evaluation at step 40000 |
| 2025-11-23 09:51:03,717 - __main__ - INFO - |
| 📤 Pushing checkpoint 40000 to HuggingFace Hub... |
| 2025-11-23 09:51:03,718 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 09:51:16,631 - __main__ - INFO - ✅ Checkpoint 40000 pushed successfully! |
| 2025-11-23 09:51:16,632 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-40000 |
| 2025-11-23 09:51:36,962 - __main__ - INFO - 📈 Epoch progress: 2.87/6 (47.9%) |
| 2025-11-23 09:51:57,063 - __main__ - INFO - 📈 Epoch progress: 2.88/6 (48.0%) |
| 2025-11-23 09:52:17,257 - __main__ - INFO - 📈 Epoch progress: 2.89/6 (48.1%) |
| 2025-11-23 09:52:37,641 - __main__ - INFO - 📈 Epoch progress: 2.90/6 (48.3%) |
| 2025-11-23 09:52:58,165 - __main__ - INFO - 📈 Epoch progress: 2.90/6 (48.4%) |
| 2025-11-23 09:53:18,722 - __main__ - INFO - 📈 Epoch progress: 2.91/6 (48.5%) |
| 2025-11-23 09:53:39,389 - __main__ - INFO - 📈 Epoch progress: 2.92/6 (48.6%) |
| 2025-11-23 09:53:59,669 - __main__ - INFO - 📈 Epoch progress: 2.92/6 (48.7%) |
| 2025-11-23 09:54:20,128 - __main__ - INFO - 📈 Epoch progress: 2.93/6 (48.9%) |
| 2025-11-23 09:54:40,680 - __main__ - INFO - 📈 Epoch progress: 2.94/6 (49.0%) |
| 2025-11-23 09:55:01,174 - __main__ - INFO - 📈 Epoch progress: 2.95/6 (49.1%) |
| 2025-11-23 09:55:21,712 - __main__ - INFO - 📈 Epoch progress: 2.95/6 (49.2%) |
| 2025-11-23 09:55:42,453 - __main__ - INFO - 📈 Epoch progress: 2.96/6 (49.3%) |
| 2025-11-23 09:56:02,593 - __main__ - INFO - 📈 Epoch progress: 2.97/6 (49.5%) |
| 2025-11-23 09:56:22,738 - __main__ - INFO - 📈 Epoch progress: 2.97/6 (49.6%) |
| 2025-11-23 09:56:43,221 - __main__ - INFO - 📈 Epoch progress: 2.98/6 (49.7%) |
| 2025-11-23 09:57:03,321 - __main__ - INFO - 📈 Epoch progress: 2.99/6 (49.8%) |
| 2025-11-23 09:57:23,352 - __main__ - INFO - 📈 Epoch progress: 3.00/6 (49.9%) |
| 2025-11-23 09:57:34,917 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 09:57:46,060 - __main__ - INFO - 📈 Epoch progress: 3.00/6 (50.0%) |
| 2025-11-23 09:58:06,345 - __main__ - INFO - 📈 Epoch progress: 3.01/6 (50.2%) |
| 2025-11-23 09:58:47,114 - __main__ - INFO - 📊 Evaluation 22: BLEU = 0.1251 |
| 2025-11-23 09:58:47,436 - __main__ - INFO - 📊 Evaluation at step 42000 |
| 2025-11-23 09:58:51,182 - __main__ - INFO - |
| 📤 Pushing checkpoint 42000 to HuggingFace Hub... |
| 2025-11-23 09:58:51,183 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 09:59:02,141 - __main__ - INFO - ✅ Checkpoint 42000 pushed successfully! |
| 2025-11-23 09:59:02,145 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-42000 |
| 2025-11-23 09:59:22,082 - __main__ - INFO - 📈 Epoch progress: 3.02/6 (50.3%) |
| 2025-11-23 09:59:42,385 - __main__ - INFO - 📈 Epoch progress: 3.02/6 (50.4%) |
| 2025-11-23 10:00:02,836 - __main__ - INFO - 📈 Epoch progress: 3.03/6 (50.5%) |
| 2025-11-23 10:00:23,468 - __main__ - INFO - 📈 Epoch progress: 3.04/6 (50.6%) |
| 2025-11-23 10:00:43,639 - __main__ - INFO - 📈 Epoch progress: 3.05/6 (50.8%) |
| 2025-11-23 10:01:03,911 - __main__ - INFO - 📈 Epoch progress: 3.05/6 (50.9%) |
| 2025-11-23 10:01:24,117 - __main__ - INFO - 📈 Epoch progress: 3.06/6 (51.0%) |
| 2025-11-23 10:01:44,959 - __main__ - INFO - 📈 Epoch progress: 3.07/6 (51.1%) |
| 2025-11-23 10:02:05,692 - __main__ - INFO - 📈 Epoch progress: 3.07/6 (51.2%) |
| 2025-11-23 10:02:36,583 - __main__ - INFO - 📈 Epoch progress: 3.08/6 (51.4%) |
| 2025-11-23 10:02:57,295 - __main__ - INFO - 📈 Epoch progress: 3.09/6 (51.5%) |
| 2025-11-23 10:03:17,602 - __main__ - INFO - 📈 Epoch progress: 3.10/6 (51.6%) |
| 2025-11-23 10:03:38,131 - __main__ - INFO - 📈 Epoch progress: 3.10/6 (51.7%) |
| 2025-11-23 10:03:58,263 - __main__ - INFO - 📈 Epoch progress: 3.11/6 (51.8%) |
| 2025-11-23 10:04:19,115 - __main__ - INFO - 📈 Epoch progress: 3.12/6 (52.0%) |
| 2025-11-23 10:04:39,426 - __main__ - INFO - 📈 Epoch progress: 3.12/6 (52.1%) |
| 2025-11-23 10:04:59,939 - __main__ - INFO - 📈 Epoch progress: 3.13/6 (52.2%) |
| 2025-11-23 10:05:20,284 - __main__ - INFO - 📈 Epoch progress: 3.14/6 (52.3%) |
| 2025-11-23 10:05:40,762 - __main__ - INFO - 📈 Epoch progress: 3.15/6 (52.4%) |
| 2025-11-23 10:06:01,395 - __main__ - INFO - 📈 Epoch progress: 3.15/6 (52.6%) |
| 2025-11-23 10:06:41,220 - __main__ - INFO - 📊 Evaluation 23: BLEU = 0.1224 |
| 2025-11-23 10:06:41,527 - __main__ - INFO - 📊 Evaluation at step 44000 |
| 2025-11-23 10:06:45,356 - __main__ - INFO - |
| 📤 Pushing checkpoint 44000 to HuggingFace Hub... |
| 2025-11-23 10:06:45,357 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 10:06:55,213 - __main__ - INFO - ✅ Checkpoint 44000 pushed successfully! |
| 2025-11-23 10:06:55,215 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-44000 |
| 2025-11-23 10:07:15,185 - __main__ - INFO - 📈 Epoch progress: 3.16/6 (52.7%) |
| 2025-11-23 10:07:35,739 - __main__ - INFO - 📈 Epoch progress: 3.17/6 (52.8%) |
| 2025-11-23 10:07:56,198 - __main__ - INFO - 📈 Epoch progress: 3.17/6 (52.9%) |
| 2025-11-23 10:08:16,815 - __main__ - INFO - 📈 Epoch progress: 3.18/6 (53.0%) |
| 2025-11-23 10:08:37,084 - __main__ - INFO - 📈 Epoch progress: 3.19/6 (53.2%) |
| 2025-11-23 10:08:57,593 - __main__ - INFO - 📈 Epoch progress: 3.20/6 (53.3%) |
| 2025-11-23 10:09:18,413 - __main__ - INFO - 📈 Epoch progress: 3.20/6 (53.4%) |
| 2025-11-23 10:09:38,869 - __main__ - INFO - 📈 Epoch progress: 3.21/6 (53.5%) |
| 2025-11-23 10:09:58,906 - __main__ - INFO - 📈 Epoch progress: 3.22/6 (53.6%) |
| 2025-11-23 10:10:19,348 - __main__ - INFO - 📈 Epoch progress: 3.23/6 (53.8%) |
| 2025-11-23 10:10:39,588 - __main__ - INFO - 📈 Epoch progress: 3.23/6 (53.9%) |
| 2025-11-23 10:11:00,601 - __main__ - INFO - 📈 Epoch progress: 3.24/6 (54.0%) |
| 2025-11-23 10:11:21,144 - __main__ - INFO - 📈 Epoch progress: 3.25/6 (54.1%) |
| 2025-11-23 10:11:41,460 - __main__ - INFO - 📈 Epoch progress: 3.25/6 (54.2%) |
| 2025-11-23 10:12:01,734 - __main__ - INFO - 📈 Epoch progress: 3.26/6 (54.3%) |
| 2025-11-23 10:12:22,032 - __main__ - INFO - 📈 Epoch progress: 3.27/6 (54.5%) |
| 2025-11-23 10:12:42,864 - __main__ - INFO - 📈 Epoch progress: 3.28/6 (54.6%) |
| 2025-11-23 10:13:03,465 - __main__ - INFO - 📈 Epoch progress: 3.28/6 (54.7%) |
| 2025-11-23 10:13:24,120 - __main__ - INFO - 📈 Epoch progress: 3.29/6 (54.8%) |
| 2025-11-23 10:13:43,882 - __main__ - INFO - 📈 Epoch progress: 3.30/6 (54.9%) |
| 2025-11-23 10:14:27,364 - __main__ - INFO - 📊 Evaluation 24: BLEU = 0.1211 |
| 2025-11-23 10:14:27,682 - __main__ - INFO - 📊 Evaluation at step 46000 |
| 2025-11-23 10:14:31,358 - __main__ - INFO - |
| 📤 Pushing checkpoint 46000 to HuggingFace Hub... |
| 2025-11-23 10:14:31,359 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 10:14:41,433 - __main__ - INFO - ✅ Checkpoint 46000 pushed successfully! |
| 2025-11-23 10:14:41,435 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-46000 |
| 2025-11-23 10:15:01,282 - __main__ - INFO - 📈 Epoch progress: 3.30/6 (55.1%) |
| 2025-11-23 10:15:21,079 - __main__ - INFO - 📈 Epoch progress: 3.31/6 (55.2%) |
| 2025-11-23 10:15:41,403 - __main__ - INFO - 📈 Epoch progress: 3.32/6 (55.3%) |
| 2025-11-23 10:16:01,219 - __main__ - INFO - 📈 Epoch progress: 3.33/6 (55.4%) |
| 2025-11-23 10:16:21,578 - __main__ - INFO - 📈 Epoch progress: 3.33/6 (55.5%) |
| 2025-11-23 10:16:42,365 - __main__ - INFO - 📈 Epoch progress: 3.34/6 (55.7%) |
| 2025-11-23 10:17:02,514 - __main__ - INFO - 📈 Epoch progress: 3.35/6 (55.8%) |
| 2025-11-23 10:17:22,687 - __main__ - INFO - 📈 Epoch progress: 3.35/6 (55.9%) |
| 2025-11-23 10:17:43,276 - __main__ - INFO - 📈 Epoch progress: 3.36/6 (56.0%) |
| 2025-11-23 10:18:04,009 - __main__ - INFO - 📈 Epoch progress: 3.37/6 (56.1%) |
| 2025-11-23 10:18:24,324 - __main__ - INFO - 📈 Epoch progress: 3.38/6 (56.3%) |
| 2025-11-23 10:18:44,948 - __main__ - INFO - 📈 Epoch progress: 3.38/6 (56.4%) |
| 2025-11-23 10:19:04,953 - __main__ - INFO - 📈 Epoch progress: 3.39/6 (56.5%) |
| 2025-11-23 10:19:25,269 - __main__ - INFO - 📈 Epoch progress: 3.40/6 (56.6%) |
| 2025-11-23 10:19:45,740 - __main__ - INFO - 📈 Epoch progress: 3.40/6 (56.7%) |
| 2025-11-23 10:20:05,871 - __main__ - INFO - 📈 Epoch progress: 3.41/6 (56.9%) |
| 2025-11-23 10:20:26,185 - __main__ - INFO - 📈 Epoch progress: 3.42/6 (57.0%) |
| 2025-11-23 10:20:46,485 - __main__ - INFO - 📈 Epoch progress: 3.43/6 (57.1%) |
| 2025-11-23 10:21:06,588 - __main__ - INFO - 📈 Epoch progress: 3.43/6 (57.2%) |
| 2025-11-23 10:21:26,738 - __main__ - INFO - 📈 Epoch progress: 3.44/6 (57.3%) |
| 2025-11-23 10:21:58,681 - __main__ - INFO - 📊 Evaluation 25: BLEU = 0.1286 |
| 2025-11-23 10:21:58,987 - __main__ - INFO - 📊 Evaluation at step 48000 |
| 2025-11-23 10:22:03,116 - __main__ - INFO - |
| 📤 Pushing checkpoint 48000 to HuggingFace Hub... |
| 2025-11-23 10:22:03,118 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 10:22:14,031 - __main__ - INFO - ✅ Checkpoint 48000 pushed successfully! |
| 2025-11-23 10:22:14,033 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-48000 |
| 2025-11-23 10:22:34,198 - __main__ - INFO - 📈 Epoch progress: 3.45/6 (57.5%) |
| 2025-11-23 10:22:54,429 - __main__ - INFO - 📈 Epoch progress: 3.45/6 (57.6%) |
| 2025-11-23 10:23:15,280 - __main__ - INFO - 📈 Epoch progress: 3.46/6 (57.7%) |
| 2025-11-23 10:23:35,391 - __main__ - INFO - 📈 Epoch progress: 3.47/6 (57.8%) |
| 2025-11-23 10:23:55,950 - __main__ - INFO - 📈 Epoch progress: 3.48/6 (57.9%) |
| 2025-11-23 10:24:17,499 - __main__ - INFO - 📈 Epoch progress: 3.48/6 (58.1%) |
| 2025-11-23 10:24:38,058 - __main__ - INFO - 📈 Epoch progress: 3.49/6 (58.2%) |
| 2025-11-23 10:24:57,657 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:25:07,623 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:25:07,623 - __main__ - INFO - 📈 Epoch progress: 3.50/6 (58.3%) |
| 2025-11-23 10:25:07,955 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:25:17,775 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:25:28,268 - __main__ - INFO - 📈 Epoch progress: 3.50/6 (58.4%) |
| 2025-11-23 10:25:48,893 - __main__ - INFO - 📈 Epoch progress: 3.51/6 (58.5%) |
| 2025-11-23 10:26:09,380 - __main__ - INFO - 📈 Epoch progress: 3.52/6 (58.6%) |
| 2025-11-23 10:26:29,817 - __main__ - INFO - 📈 Epoch progress: 3.53/6 (58.8%) |
| 2025-11-23 10:26:50,163 - __main__ - INFO - 📈 Epoch progress: 3.53/6 (58.9%) |
| 2025-11-23 10:27:10,420 - __main__ - INFO - 📈 Epoch progress: 3.54/6 (59.0%) |
| 2025-11-23 10:27:30,882 - __main__ - INFO - 📈 Epoch progress: 3.55/6 (59.1%) |
| 2025-11-23 10:27:50,942 - __main__ - INFO - 📈 Epoch progress: 3.55/6 (59.2%) |
| 2025-11-23 10:28:11,096 - __main__ - INFO - 📈 Epoch progress: 3.56/6 (59.4%) |
| 2025-11-23 10:28:31,403 - __main__ - INFO - 📈 Epoch progress: 3.57/6 (59.5%) |
| 2025-11-23 10:28:51,757 - __main__ - INFO - 📈 Epoch progress: 3.58/6 (59.6%) |
| 2025-11-23 10:29:12,648 - __main__ - INFO - 📈 Epoch progress: 3.58/6 (59.7%) |
| 2025-11-23 10:29:49,189 - __main__ - INFO - 📊 Evaluation 26: BLEU = 0.1268 |
| 2025-11-23 10:29:49,496 - __main__ - INFO - 📊 Evaluation at step 50000 |
| 2025-11-23 10:29:53,161 - __main__ - INFO - |
| 📤 Pushing checkpoint 50000 to HuggingFace Hub... |
| 2025-11-23 10:29:53,165 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 10:30:05,379 - __main__ - INFO - ✅ Checkpoint 50000 pushed successfully! |
| 2025-11-23 10:30:05,381 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-50000 |
| 2025-11-23 10:30:25,789 - __main__ - INFO - 📈 Epoch progress: 3.59/6 (59.8%) |
| 2025-11-23 10:30:46,522 - __main__ - INFO - 📈 Epoch progress: 3.60/6 (60.0%) |
| 2025-11-23 10:31:07,001 - __main__ - INFO - 📈 Epoch progress: 3.60/6 (60.1%) |
| 2025-11-23 10:31:27,491 - __main__ - INFO - 📈 Epoch progress: 3.61/6 (60.2%) |
| 2025-11-23 10:31:48,431 - __main__ - INFO - 📈 Epoch progress: 3.62/6 (60.3%) |
| 2025-11-23 10:32:08,645 - __main__ - INFO - 📈 Epoch progress: 3.63/6 (60.4%) |
| 2025-11-23 10:32:28,921 - __main__ - INFO - 📈 Epoch progress: 3.63/6 (60.6%) |
| 2025-11-23 10:32:49,144 - __main__ - INFO - 📈 Epoch progress: 3.64/6 (60.7%) |
| 2025-11-23 10:33:09,558 - __main__ - INFO - 📈 Epoch progress: 3.65/6 (60.8%) |
| 2025-11-23 10:33:29,850 - __main__ - INFO - 📈 Epoch progress: 3.66/6 (60.9%) |
| 2025-11-23 10:33:41,292 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:33:51,495 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:33:51,495 - __main__ - INFO - 📈 Epoch progress: 3.66/6 (61.0%) |
| 2025-11-23 10:33:51,820 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:34:12,028 - __main__ - INFO - 📈 Epoch progress: 3.67/6 (61.2%) |
| 2025-11-23 10:34:32,306 - __main__ - INFO - 📈 Epoch progress: 3.68/6 (61.3%) |
| 2025-11-23 10:34:52,494 - __main__ - INFO - 📈 Epoch progress: 3.68/6 (61.4%) |
| 2025-11-23 10:35:12,710 - __main__ - INFO - 📈 Epoch progress: 3.69/6 (61.5%) |
| 2025-11-23 10:35:33,230 - __main__ - INFO - 📈 Epoch progress: 3.70/6 (61.6%) |
| 2025-11-23 10:35:54,191 - __main__ - INFO - 📈 Epoch progress: 3.71/6 (61.8%) |
| 2025-11-23 10:36:14,739 - __main__ - INFO - 📈 Epoch progress: 3.71/6 (61.9%) |
| 2025-11-23 10:36:35,317 - __main__ - INFO - 📈 Epoch progress: 3.72/6 (62.0%) |
| 2025-11-23 10:36:55,536 - __main__ - INFO - 📈 Epoch progress: 3.73/6 (62.1%) |
| 2025-11-23 10:37:31,061 - __main__ - INFO - 📊 Evaluation 27: BLEU = 0.1224 |
| 2025-11-23 10:37:31,395 - __main__ - INFO - 📊 Evaluation at step 52000 |
| 2025-11-23 10:37:34,852 - __main__ - INFO - |
| 📤 Pushing checkpoint 52000 to HuggingFace Hub... |
| 2025-11-23 10:37:34,852 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 10:37:44,205 - __main__ - INFO - ✅ Checkpoint 52000 pushed successfully! |
| 2025-11-23 10:37:44,216 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-52000 |
| 2025-11-23 10:38:04,587 - __main__ - INFO - 📈 Epoch progress: 3.73/6 (62.2%) |
| 2025-11-23 10:38:24,782 - __main__ - INFO - 📈 Epoch progress: 3.74/6 (62.4%) |
| 2025-11-23 10:38:45,142 - __main__ - INFO - 📈 Epoch progress: 3.75/6 (62.5%) |
| 2025-11-23 10:39:06,140 - __main__ - INFO - 📈 Epoch progress: 3.76/6 (62.6%) |
| 2025-11-23 10:39:26,435 - __main__ - INFO - 📈 Epoch progress: 3.76/6 (62.7%) |
| 2025-11-23 10:39:46,470 - __main__ - INFO - 📈 Epoch progress: 3.77/6 (62.8%) |
| 2025-11-23 10:40:06,885 - __main__ - INFO - 📈 Epoch progress: 3.78/6 (62.9%) |
| 2025-11-23 10:40:28,164 - __main__ - INFO - 📈 Epoch progress: 3.78/6 (63.1%) |
| 2025-11-23 10:40:48,905 - __main__ - INFO - 📈 Epoch progress: 3.79/6 (63.2%) |
| 2025-11-23 10:41:09,341 - __main__ - INFO - 📈 Epoch progress: 3.80/6 (63.3%) |
| 2025-11-23 10:41:29,882 - __main__ - INFO - 📈 Epoch progress: 3.81/6 (63.4%) |
| 2025-11-23 10:41:50,995 - __main__ - INFO - 📈 Epoch progress: 3.81/6 (63.5%) |
| 2025-11-23 10:42:11,398 - __main__ - INFO - 📈 Epoch progress: 3.82/6 (63.7%) |
| 2025-11-23 10:42:31,737 - __main__ - INFO - 📈 Epoch progress: 3.83/6 (63.8%) |
| 2025-11-23 10:42:53,318 - __main__ - INFO - 📈 Epoch progress: 3.83/6 (63.9%) |
| 2025-11-23 10:43:13,652 - __main__ - INFO - 📈 Epoch progress: 3.84/6 (64.0%) |
| 2025-11-23 10:43:33,771 - __main__ - INFO - 📈 Epoch progress: 3.85/6 (64.1%) |
| 2025-11-23 10:43:54,275 - __main__ - INFO - 📈 Epoch progress: 3.86/6 (64.3%) |
| 2025-11-23 10:44:15,406 - __main__ - INFO - 📈 Epoch progress: 3.86/6 (64.4%) |
| 2025-11-23 10:44:35,479 - __main__ - INFO - 📈 Epoch progress: 3.87/6 (64.5%) |
| 2025-11-23 10:45:07,040 - __main__ - INFO - 📊 Evaluation 28: BLEU = 0.1251 |
| 2025-11-23 10:45:07,376 - __main__ - INFO - 📊 Evaluation at step 54000 |
| 2025-11-23 10:45:11,318 - __main__ - INFO - |
| 📤 Pushing checkpoint 54000 to HuggingFace Hub... |
| 2025-11-23 10:45:11,319 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 10:45:20,609 - __main__ - INFO - ✅ Checkpoint 54000 pushed successfully! |
| 2025-11-23 10:45:20,619 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-54000 |
| 2025-11-23 10:45:40,565 - __main__ - INFO - 📈 Epoch progress: 3.88/6 (64.6%) |
| 2025-11-23 10:46:00,991 - __main__ - INFO - 📈 Epoch progress: 3.88/6 (64.7%) |
| 2025-11-23 10:46:21,867 - __main__ - INFO - 📈 Epoch progress: 3.89/6 (64.9%) |
| 2025-11-23 10:46:42,776 - __main__ - INFO - 📈 Epoch progress: 3.90/6 (65.0%) |
| 2025-11-23 10:47:03,407 - __main__ - INFO - 📈 Epoch progress: 3.91/6 (65.1%) |
| 2025-11-23 10:47:24,292 - __main__ - INFO - 📈 Epoch progress: 3.91/6 (65.2%) |
| 2025-11-23 10:47:44,906 - __main__ - INFO - 📈 Epoch progress: 3.92/6 (65.3%) |
| 2025-11-23 10:48:05,505 - __main__ - INFO - 📈 Epoch progress: 3.93/6 (65.5%) |
| 2025-11-23 10:48:25,824 - __main__ - INFO - 📈 Epoch progress: 3.93/6 (65.6%) |
| 2025-11-23 10:48:46,280 - __main__ - INFO - 📈 Epoch progress: 3.94/6 (65.7%) |
| 2025-11-23 10:49:06,984 - __main__ - INFO - 📈 Epoch progress: 3.95/6 (65.8%) |
| 2025-11-23 10:49:27,677 - __main__ - INFO - 📈 Epoch progress: 3.96/6 (65.9%) |
| 2025-11-23 10:49:47,961 - __main__ - INFO - 📈 Epoch progress: 3.96/6 (66.1%) |
| 2025-11-23 10:50:08,291 - __main__ - INFO - 📈 Epoch progress: 3.97/6 (66.2%) |
| 2025-11-23 10:50:29,023 - __main__ - INFO - 📈 Epoch progress: 3.98/6 (66.3%) |
| 2025-11-23 10:50:49,463 - __main__ - INFO - 📈 Epoch progress: 3.98/6 (66.4%) |
| 2025-11-23 10:51:09,741 - __main__ - INFO - 📈 Epoch progress: 3.99/6 (66.5%) |
| 2025-11-23 10:51:26,974 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:51:37,118 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:51:37,119 - __main__ - INFO - 📈 Epoch progress: 4.00/6 (66.7%) |
| 2025-11-23 10:51:37,947 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:51:49,497 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:51:59,344 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:51:59,346 - __main__ - INFO - 📈 Epoch progress: 4.01/6 (66.8%) |
| 2025-11-23 10:51:59,640 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:52:19,475 - __main__ - INFO - 📈 Epoch progress: 4.01/6 (66.9%) |
| 2025-11-23 10:52:52,564 - __main__ - INFO - 📊 Evaluation 29: BLEU = 0.1264 |
| 2025-11-23 10:52:52,906 - __main__ - INFO - 📊 Evaluation at step 56000 |
| 2025-11-23 10:52:57,365 - __main__ - INFO - |
| 📤 Pushing checkpoint 56000 to HuggingFace Hub... |
| 2025-11-23 10:52:57,366 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 10:53:07,429 - __main__ - INFO - ✅ Checkpoint 56000 pushed successfully! |
| 2025-11-23 10:53:07,430 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-56000 |
| 2025-11-23 10:53:27,780 - __main__ - INFO - 📈 Epoch progress: 4.02/6 (67.0%) |
| 2025-11-23 10:53:48,060 - __main__ - INFO - 📈 Epoch progress: 4.03/6 (67.1%) |
| 2025-11-23 10:54:08,277 - __main__ - INFO - 📈 Epoch progress: 4.03/6 (67.2%) |
| 2025-11-23 10:54:28,768 - __main__ - INFO - 📈 Epoch progress: 4.04/6 (67.4%) |
| 2025-11-23 10:54:49,715 - __main__ - INFO - 📈 Epoch progress: 4.05/6 (67.5%) |
| 2025-11-23 10:55:10,410 - __main__ - INFO - 📈 Epoch progress: 4.06/6 (67.6%) |
| 2025-11-23 10:55:30,909 - __main__ - INFO - 📈 Epoch progress: 4.06/6 (67.7%) |
| 2025-11-23 10:55:51,172 - __main__ - INFO - 📈 Epoch progress: 4.07/6 (67.8%) |
| 2025-11-23 10:56:11,448 - __main__ - INFO - 📈 Epoch progress: 4.08/6 (68.0%) |
| 2025-11-23 10:56:26,100 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:56:36,167 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:56:36,170 - __main__ - INFO - 📈 Epoch progress: 4.09/6 (68.1%) |
| 2025-11-23 10:56:36,457 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:56:46,422 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:56:56,244 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:56:56,249 - __main__ - INFO - 📈 Epoch progress: 4.09/6 (68.2%) |
| 2025-11-23 10:56:56,549 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 10:57:16,666 - __main__ - INFO - 📈 Epoch progress: 4.10/6 (68.3%) |
| 2025-11-23 10:57:37,502 - __main__ - INFO - 📈 Epoch progress: 4.11/6 (68.4%) |
| 2025-11-23 10:57:58,075 - __main__ - INFO - 📈 Epoch progress: 4.11/6 (68.6%) |
| 2025-11-23 10:58:18,500 - __main__ - INFO - 📈 Epoch progress: 4.12/6 (68.7%) |
| 2025-11-23 10:58:38,963 - __main__ - INFO - 📈 Epoch progress: 4.13/6 (68.8%) |
| 2025-11-23 10:58:59,536 - __main__ - INFO - 📈 Epoch progress: 4.14/6 (68.9%) |
| 2025-11-23 10:59:20,668 - __main__ - INFO - 📈 Epoch progress: 4.14/6 (69.0%) |
| 2025-11-23 10:59:41,543 - __main__ - INFO - 📈 Epoch progress: 4.15/6 (69.2%) |
| 2025-11-23 11:00:02,497 - __main__ - INFO - 📈 Epoch progress: 4.16/6 (69.3%) |
| 2025-11-23 11:00:39,692 - __main__ - INFO - 📊 Evaluation 30: BLEU = 0.1249 |
| 2025-11-23 11:00:40,019 - __main__ - INFO - 📊 Evaluation at step 58000 |
| 2025-11-23 11:00:43,865 - __main__ - INFO - |
| 📤 Pushing checkpoint 58000 to HuggingFace Hub... |
| 2025-11-23 11:00:43,866 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 11:00:53,580 - __main__ - INFO - ✅ Checkpoint 58000 pushed successfully! |
| 2025-11-23 11:00:53,581 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-58000 |
| 2025-11-23 11:01:14,112 - __main__ - INFO - 📈 Epoch progress: 4.16/6 (69.4%) |
| 2025-11-23 11:01:34,296 - __main__ - INFO - 📈 Epoch progress: 4.17/6 (69.5%) |
| 2025-11-23 11:01:55,224 - __main__ - INFO - 📈 Epoch progress: 4.18/6 (69.6%) |
| 2025-11-23 11:02:15,819 - __main__ - INFO - 📈 Epoch progress: 4.19/6 (69.8%) |
| 2025-11-23 11:02:36,025 - __main__ - INFO - 📈 Epoch progress: 4.19/6 (69.9%) |
| 2025-11-23 11:02:56,860 - __main__ - INFO - 📈 Epoch progress: 4.20/6 (70.0%) |
| 2025-11-23 11:03:17,581 - __main__ - INFO - 📈 Epoch progress: 4.21/6 (70.1%) |
| 2025-11-23 11:03:38,145 - __main__ - INFO - 📈 Epoch progress: 4.21/6 (70.2%) |
| 2025-11-23 11:03:59,166 - __main__ - INFO - 📈 Epoch progress: 4.22/6 (70.4%) |
| 2025-11-23 11:04:19,583 - __main__ - INFO - 📈 Epoch progress: 4.23/6 (70.5%) |
| 2025-11-23 11:04:40,498 - __main__ - INFO - 📈 Epoch progress: 4.24/6 (70.6%) |
| 2025-11-23 11:05:01,826 - __main__ - INFO - 📈 Epoch progress: 4.24/6 (70.7%) |
| 2025-11-23 11:05:22,634 - __main__ - INFO - 📈 Epoch progress: 4.25/6 (70.8%) |
| 2025-11-23 11:05:43,338 - __main__ - INFO - 📈 Epoch progress: 4.26/6 (71.0%) |
| 2025-11-23 11:06:04,202 - __main__ - INFO - 📈 Epoch progress: 4.26/6 (71.1%) |
| 2025-11-23 11:06:25,149 - __main__ - INFO - 📈 Epoch progress: 4.27/6 (71.2%) |
| 2025-11-23 11:06:45,679 - __main__ - INFO - 📈 Epoch progress: 4.28/6 (71.3%) |
| 2025-11-23 11:07:06,224 - __main__ - INFO - 📈 Epoch progress: 4.29/6 (71.4%) |
| 2025-11-23 11:07:26,901 - __main__ - INFO - 📈 Epoch progress: 4.29/6 (71.5%) |
| 2025-11-23 11:07:47,653 - __main__ - INFO - 📈 Epoch progress: 4.30/6 (71.7%) |
| 2025-11-23 11:08:19,925 - __main__ - INFO - 📊 Evaluation 31: BLEU = 0.1275 |
| 2025-11-23 11:08:20,305 - __main__ - INFO - 📊 Evaluation at step 60000 |
| 2025-11-23 11:08:25,271 - __main__ - INFO - |
| 📤 Pushing checkpoint 60000 to HuggingFace Hub... |
| 2025-11-23 11:08:25,272 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 11:08:35,661 - __main__ - INFO - ✅ Checkpoint 60000 pushed successfully! |
| 2025-11-23 11:08:35,662 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-60000 |
| 2025-11-23 11:08:55,711 - __main__ - INFO - 📈 Epoch progress: 4.31/6 (71.8%) |
| 2025-11-23 11:09:16,419 - __main__ - INFO - 📈 Epoch progress: 4.31/6 (71.9%) |
| 2025-11-23 11:09:36,830 - __main__ - INFO - 📈 Epoch progress: 4.32/6 (72.0%) |
| 2025-11-23 11:09:57,754 - __main__ - INFO - 📈 Epoch progress: 4.33/6 (72.1%) |
| 2025-11-23 11:10:18,572 - __main__ - INFO - 📈 Epoch progress: 4.34/6 (72.3%) |
| 2025-11-23 11:10:38,994 - __main__ - INFO - 📈 Epoch progress: 4.34/6 (72.4%) |
| 2025-11-23 11:10:59,827 - __main__ - INFO - 📈 Epoch progress: 4.35/6 (72.5%) |
| 2025-11-23 11:11:20,341 - __main__ - INFO - 📈 Epoch progress: 4.36/6 (72.6%) |
| 2025-11-23 11:11:41,034 - __main__ - INFO - 📈 Epoch progress: 4.36/6 (72.7%) |
| 2025-11-23 11:12:01,805 - __main__ - INFO - 📈 Epoch progress: 4.37/6 (72.9%) |
| 2025-11-23 11:12:22,317 - __main__ - INFO - 📈 Epoch progress: 4.38/6 (73.0%) |
| 2025-11-23 11:12:42,707 - __main__ - INFO - 📈 Epoch progress: 4.39/6 (73.1%) |
| 2025-11-23 11:13:03,904 - __main__ - INFO - 📈 Epoch progress: 4.39/6 (73.2%) |
| 2025-11-23 11:13:24,153 - __main__ - INFO - 📈 Epoch progress: 4.40/6 (73.3%) |
| 2025-11-23 11:13:44,993 - __main__ - INFO - 📈 Epoch progress: 4.41/6 (73.5%) |
| 2025-11-23 11:14:05,562 - __main__ - INFO - 📈 Epoch progress: 4.41/6 (73.6%) |
| 2025-11-23 11:14:25,940 - __main__ - INFO - 📈 Epoch progress: 4.42/6 (73.7%) |
| 2025-11-23 11:14:55,966 - __main__ - INFO - 📈 Epoch progress: 4.43/6 (73.8%) |
| 2025-11-23 11:15:16,244 - __main__ - INFO - 📈 Epoch progress: 4.44/6 (73.9%) |
| 2025-11-23 11:15:36,765 - __main__ - INFO - 📈 Epoch progress: 4.44/6 (74.1%) |
| 2025-11-23 11:16:13,684 - __main__ - INFO - 📊 Evaluation 32: BLEU = 0.1237 |
| 2025-11-23 11:16:14,117 - __main__ - INFO - 📊 Evaluation at step 62000 |
| 2025-11-23 11:16:18,395 - __main__ - INFO - |
| 📤 Pushing checkpoint 62000 to HuggingFace Hub... |
| 2025-11-23 11:16:18,396 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 11:16:28,741 - __main__ - INFO - ✅ Checkpoint 62000 pushed successfully! |
| 2025-11-23 11:16:28,743 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-62000 |
| 2025-11-23 11:16:49,119 - __main__ - INFO - 📈 Epoch progress: 4.45/6 (74.2%) |
| 2025-11-23 11:17:09,729 - __main__ - INFO - 📈 Epoch progress: 4.46/6 (74.3%) |
| 2025-11-23 11:17:30,183 - __main__ - INFO - 📈 Epoch progress: 4.46/6 (74.4%) |
| 2025-11-23 11:17:50,985 - __main__ - INFO - 📈 Epoch progress: 4.47/6 (74.5%) |
| 2025-11-23 11:18:11,757 - __main__ - INFO - 📈 Epoch progress: 4.48/6 (74.7%) |
| 2025-11-23 11:18:32,308 - __main__ - INFO - 📈 Epoch progress: 4.49/6 (74.8%) |
| 2025-11-23 11:18:53,006 - __main__ - INFO - 📈 Epoch progress: 4.49/6 (74.9%) |
| 2025-11-23 11:19:13,448 - __main__ - INFO - 📈 Epoch progress: 4.50/6 (75.0%) |
| 2025-11-23 11:19:33,723 - __main__ - INFO - 📈 Epoch progress: 4.51/6 (75.1%) |
| 2025-11-23 11:19:54,909 - __main__ - INFO - 📈 Epoch progress: 4.52/6 (75.3%) |
| 2025-11-23 11:20:15,364 - __main__ - INFO - 📈 Epoch progress: 4.52/6 (75.4%) |
| 2025-11-23 11:20:35,533 - __main__ - INFO - 📈 Epoch progress: 4.53/6 (75.5%) |
| 2025-11-23 11:20:55,811 - __main__ - INFO - 📈 Epoch progress: 4.54/6 (75.6%) |
| 2025-11-23 11:21:16,256 - __main__ - INFO - 📈 Epoch progress: 4.54/6 (75.7%) |
| 2025-11-23 11:21:37,348 - __main__ - INFO - 📈 Epoch progress: 4.55/6 (75.8%) |
| 2025-11-23 11:21:57,843 - __main__ - INFO - 📈 Epoch progress: 4.56/6 (76.0%) |
| 2025-11-23 11:22:18,570 - __main__ - INFO - 📈 Epoch progress: 4.57/6 (76.1%) |
| 2025-11-23 11:22:37,004 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:22:47,122 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:22:47,123 - __main__ - INFO - 📈 Epoch progress: 4.57/6 (76.2%) |
| 2025-11-23 11:22:47,478 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:23:08,019 - __main__ - INFO - 📈 Epoch progress: 4.58/6 (76.3%) |
| 2025-11-23 11:23:28,506 - __main__ - INFO - 📈 Epoch progress: 4.59/6 (76.4%) |
| 2025-11-23 11:24:04,349 - __main__ - INFO - 📊 Evaluation 33: BLEU = 0.1257 |
| 2025-11-23 11:24:04,671 - __main__ - INFO - 📊 Evaluation at step 64000 |
| 2025-11-23 11:24:08,153 - __main__ - INFO - |
| 📤 Pushing checkpoint 64000 to HuggingFace Hub... |
| 2025-11-23 11:24:08,154 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 11:24:18,528 - __main__ - INFO - ✅ Checkpoint 64000 pushed successfully! |
| 2025-11-23 11:24:18,530 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-64000 |
| 2025-11-23 11:24:38,954 - __main__ - INFO - 📈 Epoch progress: 4.59/6 (76.6%) |
| 2025-11-23 11:24:59,188 - __main__ - INFO - 📈 Epoch progress: 4.60/6 (76.7%) |
| 2025-11-23 11:25:19,820 - __main__ - INFO - 📈 Epoch progress: 4.61/6 (76.8%) |
| 2025-11-23 11:25:40,061 - __main__ - INFO - 📈 Epoch progress: 4.62/6 (76.9%) |
| 2025-11-23 11:26:00,309 - __main__ - INFO - 📈 Epoch progress: 4.62/6 (77.0%) |
| 2025-11-23 11:26:20,678 - __main__ - INFO - 📈 Epoch progress: 4.63/6 (77.2%) |
| 2025-11-23 11:26:41,241 - __main__ - INFO - 📈 Epoch progress: 4.64/6 (77.3%) |
| 2025-11-23 11:27:01,665 - __main__ - INFO - 📈 Epoch progress: 4.64/6 (77.4%) |
| 2025-11-23 11:27:22,445 - __main__ - INFO - 📈 Epoch progress: 4.65/6 (77.5%) |
| 2025-11-23 11:27:42,660 - __main__ - INFO - 📈 Epoch progress: 4.66/6 (77.6%) |
| 2025-11-23 11:28:02,981 - __main__ - INFO - 📈 Epoch progress: 4.67/6 (77.8%) |
| 2025-11-23 11:28:23,074 - __main__ - INFO - 📈 Epoch progress: 4.67/6 (77.9%) |
| 2025-11-23 11:28:43,752 - __main__ - INFO - 📈 Epoch progress: 4.68/6 (78.0%) |
| 2025-11-23 11:29:04,067 - __main__ - INFO - 📈 Epoch progress: 4.69/6 (78.1%) |
| 2025-11-23 11:29:24,386 - __main__ - INFO - 📈 Epoch progress: 4.69/6 (78.2%) |
| 2025-11-23 11:29:45,037 - __main__ - INFO - 📈 Epoch progress: 4.70/6 (78.4%) |
| 2025-11-23 11:30:05,369 - __main__ - INFO - 📈 Epoch progress: 4.71/6 (78.5%) |
| 2025-11-23 11:30:25,784 - __main__ - INFO - 📈 Epoch progress: 4.72/6 (78.6%) |
| 2025-11-23 11:30:46,081 - __main__ - INFO - 📈 Epoch progress: 4.72/6 (78.7%) |
| 2025-11-23 11:31:06,417 - __main__ - INFO - 📈 Epoch progress: 4.73/6 (78.8%) |
| 2025-11-23 11:31:42,452 - __main__ - INFO - 📊 Evaluation 34: BLEU = 0.1273 |
| 2025-11-23 11:31:42,774 - __main__ - INFO - 📊 Evaluation at step 66000 |
| 2025-11-23 11:31:47,184 - __main__ - INFO - |
| 📤 Pushing checkpoint 66000 to HuggingFace Hub... |
| 2025-11-23 11:31:47,185 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 11:31:57,845 - __main__ - INFO - ✅ Checkpoint 66000 pushed successfully! |
| 2025-11-23 11:31:57,846 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-66000 |
| 2025-11-23 11:32:17,970 - __main__ - INFO - 📈 Epoch progress: 4.74/6 (79.0%) |
| 2025-11-23 11:32:39,352 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:32:49,208 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:32:49,220 - __main__ - INFO - 📈 Epoch progress: 4.74/6 (79.1%) |
| 2025-11-23 11:32:49,552 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:32:59,539 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:33:09,376 - __main__ - INFO - 📈 Epoch progress: 4.75/6 (79.2%) |
| 2025-11-23 11:33:29,665 - __main__ - INFO - 📈 Epoch progress: 4.76/6 (79.3%) |
| 2025-11-23 11:33:50,740 - __main__ - INFO - 📈 Epoch progress: 4.77/6 (79.4%) |
| 2025-11-23 11:34:11,309 - __main__ - INFO - 📈 Epoch progress: 4.77/6 (79.6%) |
| 2025-11-23 11:34:31,544 - __main__ - INFO - 📈 Epoch progress: 4.78/6 (79.7%) |
| 2025-11-23 11:34:51,889 - __main__ - INFO - 📈 Epoch progress: 4.79/6 (79.8%) |
| 2025-11-23 11:35:12,416 - __main__ - INFO - 📈 Epoch progress: 4.79/6 (79.9%) |
| 2025-11-23 11:35:32,846 - __main__ - INFO - 📈 Epoch progress: 4.80/6 (80.0%) |
| 2025-11-23 11:35:53,172 - __main__ - INFO - 📈 Epoch progress: 4.81/6 (80.2%) |
| 2025-11-23 11:36:13,603 - __main__ - INFO - 📈 Epoch progress: 4.82/6 (80.3%) |
| 2025-11-23 11:36:34,105 - __main__ - INFO - 📈 Epoch progress: 4.82/6 (80.4%) |
| 2025-11-23 11:36:55,072 - __main__ - INFO - 📈 Epoch progress: 4.83/6 (80.5%) |
| 2025-11-23 11:37:15,470 - __main__ - INFO - 📈 Epoch progress: 4.84/6 (80.6%) |
| 2025-11-23 11:37:38,273 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:37:38,274 - __main__ - INFO - 📈 Epoch progress: 4.84/6 (80.7%) |
| 2025-11-23 11:37:39,127 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:37:59,322 - __main__ - INFO - 📈 Epoch progress: 4.85/6 (80.9%) |
| 2025-11-23 11:38:20,306 - __main__ - INFO - 📈 Epoch progress: 4.86/6 (81.0%) |
| 2025-11-23 11:38:41,021 - __main__ - INFO - 📈 Epoch progress: 4.87/6 (81.1%) |
| 2025-11-23 11:39:01,832 - __main__ - INFO - 📈 Epoch progress: 4.87/6 (81.2%) |
| 2025-11-23 11:39:41,982 - __main__ - INFO - 📊 Evaluation 35: BLEU = 0.1258 |
| 2025-11-23 11:39:42,313 - __main__ - INFO - 📊 Evaluation at step 68000 |
| 2025-11-23 11:39:46,566 - __main__ - INFO - |
| 📤 Pushing checkpoint 68000 to HuggingFace Hub... |
| 2025-11-23 11:39:46,567 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 11:40:01,175 - __main__ - INFO - ✅ Checkpoint 68000 pushed successfully! |
| 2025-11-23 11:40:01,176 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-68000 |
| 2025-11-23 11:40:20,875 - __main__ - INFO - 📈 Epoch progress: 4.88/6 (81.3%) |
| 2025-11-23 11:40:41,158 - __main__ - INFO - 📈 Epoch progress: 4.89/6 (81.5%) |
| 2025-11-23 11:41:01,674 - __main__ - INFO - 📈 Epoch progress: 4.90/6 (81.6%) |
| 2025-11-23 11:41:21,984 - __main__ - INFO - 📈 Epoch progress: 4.90/6 (81.7%) |
| 2025-11-23 11:41:42,193 - __main__ - INFO - 📈 Epoch progress: 4.91/6 (81.8%) |
| 2025-11-23 11:42:02,598 - __main__ - INFO - 📈 Epoch progress: 4.92/6 (81.9%) |
| 2025-11-23 11:42:22,862 - __main__ - INFO - 📈 Epoch progress: 4.92/6 (82.1%) |
| 2025-11-23 11:42:43,564 - __main__ - INFO - 📈 Epoch progress: 4.93/6 (82.2%) |
| 2025-11-23 11:43:04,912 - __main__ - INFO - 📈 Epoch progress: 4.94/6 (82.3%) |
| 2025-11-23 11:43:25,907 - __main__ - INFO - 📈 Epoch progress: 4.95/6 (82.4%) |
| 2025-11-23 11:43:46,200 - __main__ - INFO - 📈 Epoch progress: 4.95/6 (82.5%) |
| 2025-11-23 11:44:06,657 - __main__ - INFO - 📈 Epoch progress: 4.96/6 (82.7%) |
| 2025-11-23 11:44:27,120 - __main__ - INFO - 📈 Epoch progress: 4.97/6 (82.8%) |
| 2025-11-23 11:44:55,607 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:44:55,616 - __main__ - INFO - 📈 Epoch progress: 4.97/6 (82.9%) |
| 2025-11-23 11:44:55,941 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:45:16,408 - __main__ - INFO - 📈 Epoch progress: 4.98/6 (83.0%) |
| 2025-11-23 11:45:36,998 - __main__ - INFO - 📈 Epoch progress: 4.99/6 (83.1%) |
| 2025-11-23 11:45:57,331 - __main__ - INFO - 📈 Epoch progress: 5.00/6 (83.3%) |
| 2025-11-23 11:46:18,794 - __main__ - INFO - 📈 Epoch progress: 5.00/6 (83.4%) |
| 2025-11-23 11:46:38,903 - __main__ - INFO - 📈 Epoch progress: 5.01/6 (83.5%) |
| 2025-11-23 11:46:59,014 - __main__ - INFO - 📈 Epoch progress: 5.02/6 (83.6%) |
| 2025-11-23 11:47:38,024 - __main__ - INFO - 📊 Evaluation 36: BLEU = 0.1254 |
| 2025-11-23 11:47:38,335 - __main__ - INFO - 📊 Evaluation at step 70000 |
| 2025-11-23 11:47:42,852 - __main__ - INFO - |
| 📤 Pushing checkpoint 70000 to HuggingFace Hub... |
| 2025-11-23 11:47:42,852 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 11:47:53,672 - __main__ - INFO - ✅ Checkpoint 70000 pushed successfully! |
| 2025-11-23 11:47:53,673 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-70000 |
| 2025-11-23 11:48:13,512 - __main__ - INFO - 📈 Epoch progress: 5.02/6 (83.7%) |
| 2025-11-23 11:48:33,762 - __main__ - INFO - 📈 Epoch progress: 5.03/6 (83.9%) |
| 2025-11-23 11:48:53,996 - __main__ - INFO - 📈 Epoch progress: 5.04/6 (84.0%) |
| 2025-11-23 11:49:14,612 - __main__ - INFO - 📈 Epoch progress: 5.05/6 (84.1%) |
| 2025-11-23 11:49:34,923 - __main__ - INFO - 📈 Epoch progress: 5.05/6 (84.2%) |
| 2025-11-23 11:49:55,370 - __main__ - INFO - 📈 Epoch progress: 5.06/6 (84.3%) |
| 2025-11-23 11:50:16,078 - __main__ - INFO - 📈 Epoch progress: 5.07/6 (84.5%) |
| 2025-11-23 11:50:36,749 - __main__ - INFO - 📈 Epoch progress: 5.07/6 (84.6%) |
| 2025-11-23 11:50:57,642 - __main__ - INFO - 📈 Epoch progress: 5.08/6 (84.7%) |
| 2025-11-23 11:51:18,248 - __main__ - INFO - 📈 Epoch progress: 5.09/6 (84.8%) |
| 2025-11-23 11:51:38,487 - __main__ - INFO - 📈 Epoch progress: 5.10/6 (84.9%) |
| 2025-11-23 11:51:59,031 - __main__ - INFO - 📈 Epoch progress: 5.10/6 (85.0%) |
| 2025-11-23 11:52:19,499 - __main__ - INFO - 📈 Epoch progress: 5.11/6 (85.2%) |
| 2025-11-23 11:52:35,208 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:52:45,037 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:52:45,039 - __main__ - INFO - 📈 Epoch progress: 5.12/6 (85.3%) |
| 2025-11-23 11:52:45,388 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 11:53:05,555 - __main__ - INFO - 📈 Epoch progress: 5.12/6 (85.4%) |
| 2025-11-23 11:53:26,834 - __main__ - INFO - 📈 Epoch progress: 5.13/6 (85.5%) |
| 2025-11-23 11:53:47,887 - __main__ - INFO - 📈 Epoch progress: 5.14/6 (85.6%) |
| 2025-11-23 11:54:08,563 - __main__ - INFO - 📈 Epoch progress: 5.15/6 (85.8%) |
| 2025-11-23 11:54:29,236 - __main__ - INFO - 📈 Epoch progress: 5.15/6 (85.9%) |
| 2025-11-23 11:54:50,073 - __main__ - INFO - 📈 Epoch progress: 5.16/6 (86.0%) |
| 2025-11-23 11:55:28,304 - __main__ - INFO - 📊 Evaluation 37: BLEU = 0.1242 |
| 2025-11-23 11:55:28,655 - __main__ - INFO - 📊 Evaluation at step 72000 |
| 2025-11-23 11:55:33,076 - __main__ - INFO - |
| 📤 Pushing checkpoint 72000 to HuggingFace Hub... |
| 2025-11-23 11:55:33,078 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 11:55:43,947 - __main__ - INFO - ✅ Checkpoint 72000 pushed successfully! |
| 2025-11-23 11:55:43,950 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-72000 |
| 2025-11-23 11:56:03,792 - __main__ - INFO - 📈 Epoch progress: 5.17/6 (86.1%) |
| 2025-11-23 11:56:25,145 - __main__ - INFO - 📈 Epoch progress: 5.17/6 (86.2%) |
| 2025-11-23 11:56:45,719 - __main__ - INFO - 📈 Epoch progress: 5.18/6 (86.4%) |
| 2025-11-23 11:57:06,529 - __main__ - INFO - 📈 Epoch progress: 5.19/6 (86.5%) |
| 2025-11-23 11:57:26,528 - __main__ - INFO - 📈 Epoch progress: 5.20/6 (86.6%) |
| 2025-11-23 11:57:46,986 - __main__ - INFO - 📈 Epoch progress: 5.20/6 (86.7%) |
| 2025-11-23 11:58:07,614 - __main__ - INFO - 📈 Epoch progress: 5.21/6 (86.8%) |
| 2025-11-23 11:58:27,841 - __main__ - INFO - 📈 Epoch progress: 5.22/6 (87.0%) |
| 2025-11-23 11:58:48,392 - __main__ - INFO - 📈 Epoch progress: 5.22/6 (87.1%) |
| 2025-11-23 11:59:09,079 - __main__ - INFO - 📈 Epoch progress: 5.23/6 (87.2%) |
| 2025-11-23 11:59:30,240 - __main__ - INFO - 📈 Epoch progress: 5.24/6 (87.3%) |
| 2025-11-23 11:59:50,776 - __main__ - INFO - 📈 Epoch progress: 5.25/6 (87.4%) |
| 2025-11-23 12:00:11,234 - __main__ - INFO - 📈 Epoch progress: 5.25/6 (87.6%) |
| 2025-11-23 12:00:31,761 - __main__ - INFO - 📈 Epoch progress: 5.26/6 (87.7%) |
| 2025-11-23 12:00:52,185 - __main__ - INFO - 📈 Epoch progress: 5.27/6 (87.8%) |
| 2025-11-23 12:01:12,397 - __main__ - INFO - 📈 Epoch progress: 5.27/6 (87.9%) |
| 2025-11-23 12:01:32,731 - __main__ - INFO - 📈 Epoch progress: 5.28/6 (88.0%) |
| 2025-11-23 12:01:53,514 - __main__ - INFO - 📈 Epoch progress: 5.29/6 (88.2%) |
| 2025-11-23 12:02:13,894 - __main__ - INFO - 📈 Epoch progress: 5.30/6 (88.3%) |
| 2025-11-23 12:02:34,676 - __main__ - INFO - 📈 Epoch progress: 5.30/6 (88.4%) |
| 2025-11-23 12:03:14,033 - __main__ - INFO - 📊 Evaluation 38: BLEU = 0.1254 |
| 2025-11-23 12:03:14,380 - __main__ - INFO - 📊 Evaluation at step 74000 |
| 2025-11-23 12:03:18,718 - __main__ - INFO - |
| 📤 Pushing checkpoint 74000 to HuggingFace Hub... |
| 2025-11-23 12:03:18,720 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 12:03:30,405 - __main__ - INFO - ✅ Checkpoint 74000 pushed successfully! |
| 2025-11-23 12:03:30,415 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-74000 |
| 2025-11-23 12:03:51,901 - __main__ - INFO - 📈 Epoch progress: 5.31/6 (88.5%) |
| 2025-11-23 12:04:12,798 - __main__ - INFO - 📈 Epoch progress: 5.32/6 (88.6%) |
| 2025-11-23 12:04:33,746 - __main__ - INFO - 📈 Epoch progress: 5.33/6 (88.8%) |
| 2025-11-23 12:04:55,695 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 12:04:55,697 - __main__ - INFO - 📈 Epoch progress: 5.33/6 (88.9%) |
| 2025-11-23 12:04:56,050 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 12:05:16,766 - __main__ - INFO - 📈 Epoch progress: 5.34/6 (89.0%) |
| 2025-11-23 12:05:38,094 - __main__ - INFO - 📈 Epoch progress: 5.35/6 (89.1%) |
| 2025-11-23 12:05:59,874 - __main__ - INFO - 📈 Epoch progress: 5.35/6 (89.2%) |
| 2025-11-23 12:06:20,713 - __main__ - INFO - 📈 Epoch progress: 5.36/6 (89.3%) |
| 2025-11-23 12:06:42,241 - __main__ - INFO - 📈 Epoch progress: 5.37/6 (89.5%) |
| 2025-11-23 12:07:02,849 - __main__ - INFO - 📈 Epoch progress: 5.38/6 (89.6%) |
| 2025-11-23 12:07:23,895 - __main__ - INFO - 📈 Epoch progress: 5.38/6 (89.7%) |
| 2025-11-23 12:07:44,628 - __main__ - INFO - 📈 Epoch progress: 5.39/6 (89.8%) |
| 2025-11-23 12:08:06,418 - __main__ - INFO - 📈 Epoch progress: 5.40/6 (89.9%) |
| 2025-11-23 12:08:21,381 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 12:08:31,578 - __main__ - INFO - 📈 Epoch progress: 5.40/6 (90.1%) |
| 2025-11-23 12:08:52,890 - __main__ - INFO - 📈 Epoch progress: 5.41/6 (90.2%) |
| 2025-11-23 12:09:14,214 - __main__ - INFO - 📈 Epoch progress: 5.42/6 (90.3%) |
| 2025-11-23 12:09:35,582 - __main__ - INFO - 📈 Epoch progress: 5.43/6 (90.4%) |
| 2025-11-23 12:09:56,347 - __main__ - INFO - 📈 Epoch progress: 5.43/6 (90.5%) |
| 2025-11-23 12:10:17,782 - __main__ - INFO - 📈 Epoch progress: 5.44/6 (90.7%) |
| 2025-11-23 12:10:38,744 - __main__ - INFO - 📈 Epoch progress: 5.45/6 (90.8%) |
| 2025-11-23 12:11:19,789 - __main__ - INFO - 📊 Evaluation 39: BLEU = 0.1258 |
| 2025-11-23 12:11:20,173 - __main__ - INFO - 📊 Evaluation at step 76000 |
| 2025-11-23 12:11:24,624 - __main__ - INFO - |
| 📤 Pushing checkpoint 76000 to HuggingFace Hub... |
| 2025-11-23 12:11:24,627 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 12:11:36,029 - __main__ - INFO - ✅ Checkpoint 76000 pushed successfully! |
| 2025-11-23 12:11:36,033 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-76000 |
| 2025-11-23 12:11:57,518 - __main__ - INFO - 📈 Epoch progress: 5.45/6 (90.9%) |
| 2025-11-23 12:12:18,678 - __main__ - INFO - 📈 Epoch progress: 5.46/6 (91.0%) |
| 2025-11-23 12:12:39,291 - __main__ - INFO - 📈 Epoch progress: 5.47/6 (91.1%) |
| 2025-11-23 12:13:00,780 - __main__ - INFO - 📈 Epoch progress: 5.48/6 (91.3%) |
| 2025-11-23 12:13:21,968 - __main__ - INFO - 📈 Epoch progress: 5.48/6 (91.4%) |
| 2025-11-23 12:13:42,669 - __main__ - INFO - 📈 Epoch progress: 5.49/6 (91.5%) |
| 2025-11-23 12:14:03,609 - __main__ - INFO - 📈 Epoch progress: 5.50/6 (91.6%) |
| 2025-11-23 12:14:24,429 - __main__ - INFO - 📈 Epoch progress: 5.50/6 (91.7%) |
| 2025-11-23 12:14:45,022 - __main__ - INFO - 📈 Epoch progress: 5.51/6 (91.9%) |
| 2025-11-23 12:15:05,930 - __main__ - INFO - 📈 Epoch progress: 5.52/6 (92.0%) |
| 2025-11-23 12:15:27,222 - __main__ - INFO - 📈 Epoch progress: 5.53/6 (92.1%) |
| 2025-11-23 12:15:48,168 - __main__ - INFO - 📈 Epoch progress: 5.53/6 (92.2%) |
| 2025-11-23 12:16:09,247 - __main__ - INFO - 📈 Epoch progress: 5.54/6 (92.3%) |
| 2025-11-23 12:16:29,893 - __main__ - INFO - 📈 Epoch progress: 5.55/6 (92.5%) |
| 2025-11-23 12:16:51,206 - __main__ - INFO - 📈 Epoch progress: 5.55/6 (92.6%) |
| 2025-11-23 12:17:12,182 - __main__ - INFO - 📈 Epoch progress: 5.56/6 (92.7%) |
| 2025-11-23 12:17:38,496 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 12:17:38,498 - __main__ - INFO - 📈 Epoch progress: 5.57/6 (92.8%) |
| 2025-11-23 12:17:38,981 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 12:17:49,457 - __main__ - WARNING - ⚠️ Error logging metrics to MLflow: API request to endpoint /api/2.0/mlflow/runs/log-batch failed with error code 530 != 200. Response body: '530 |
| The origin has been unregistered from Argo Tunnel |
| ' |
| 2025-11-23 12:18:00,372 - __main__ - INFO - 📈 Epoch progress: 5.58/6 (92.9%) |
| 2025-11-23 12:18:21,453 - __main__ - INFO - 📈 Epoch progress: 5.58/6 (93.1%) |
| 2025-11-23 12:18:42,243 - __main__ - INFO - 📈 Epoch progress: 5.59/6 (93.2%) |
| 2025-11-23 12:19:19,558 - __main__ - INFO - 📊 Evaluation 40: BLEU = 0.1256 |
| 2025-11-23 12:19:19,903 - __main__ - INFO - 📊 Evaluation at step 78000 |
| 2025-11-23 12:19:24,396 - __main__ - INFO - |
| 📤 Pushing checkpoint 78000 to HuggingFace Hub... |
| 2025-11-23 12:19:24,398 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-23 12:19:35,739 - __main__ - INFO - ✅ Checkpoint 78000 pushed successfully! |
| 2025-11-23 12:19:35,741 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-78000 |
| 2025-11-23 12:19:40,328 - __main__ - INFO - ✅ MLflow: Training finished - status_code=2.0 (FINISHED) |
| 2025-11-23 12:19:40,878 - __main__ - INFO - |
| 💾 Saving model to models/finetuned-sw-en-phase2 |
| 2025-11-23 12:19:43,015 - __main__ - INFO - ✅ Training complete! |
| 2025-11-23 12:19:43,017 - __main__ - INFO - |
| 📊 STEP 3: Final Evaluation |
| 2025-11-23 12:19:43,367 - __main__ - INFO - ✅ MLflow: Final evaluation starting - status_code=3.0 |
| 2025-11-23 12:19:43,376 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 12:19:43,378 - __main__ - INFO - 📊 Running FINAL evaluation on test set... |
| 2025-11-23 12:19:43,381 - __main__ - INFO - ====================================================================== |
| 2025-11-23 12:19:43,382 - __main__ - INFO - Tokenizing test dataset... |
| 2025-11-23 12:19:45,364 - datasets.fingerprint - WARNING - Parameter 'function'=<bound method TranslationTrainer.preprocess_function of <__main__.TranslationTrainer object at 0x7717f46e3470>> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown. |
| 2025-11-23 13:31:14,215 - __main__ - INFO - 📊 Evaluation 41: BLEU = 0.1327 |
| 2025-11-23 13:31:14,636 - __main__ - INFO - 📊 Evaluation at step 78000 |
| 2025-11-23 13:31:14,792 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-23 13:31:14,838 - __main__ - INFO - Computing predictions for COMET (1000 samples, batch_size=32)... |
| 2025-11-23 13:31:14,839 - __main__ - INFO - Batch 1/32... |
| 2025-11-23 13:31:17,926 - __main__ - INFO - Batch 5/32... |
| 2025-11-23 13:31:19,046 - __main__ - INFO - Batch 10/32... |
| 2025-11-23 13:31:20,194 - __main__ - INFO - Batch 15/32... |
| 2025-11-23 13:31:21,438 - __main__ - INFO - Batch 20/32... |
| 2025-11-23 13:31:22,619 - __main__ - INFO - Batch 25/32... |
| 2025-11-23 13:31:23,790 - __main__ - INFO - Batch 30/32... |
| 2025-11-23 13:31:24,397 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-23 13:31:24,429 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-23 13:31:24,438 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-23 13:31:24,440 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-23 13:31:24,442 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-23 13:31:29,652 - __main__ - INFO - ✅ COMET-QE: 0.1811 ± 0.2659 |
| 2025-11-23 13:31:29,654 - __main__ - INFO - |
| ✅ FINAL Test Results: |
| 2025-11-23 13:31:29,655 - __main__ - INFO - BLEU: 0.1327 |
| 2025-11-23 13:31:29,655 - __main__ - INFO - chrF: 32.59 |
| 2025-11-23 13:31:29,656 - __main__ - INFO - COMET-QE: 0.1811 |
| 2025-11-23 13:31:29,656 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 13:31:29,658 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 13:31:29,658 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-23 13:31:29,659 - __main__ - INFO - ====================================================================== |
| 2025-11-23 13:31:29,660 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-23 13:31:29,661 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-23 13:31:31,728 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-23 13:31:31,796 - __main__ - INFO - ✅ Domain chrF: 1.12 |
| 2025-11-23 13:31:31,797 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-23 13:31:31,825 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-23 13:31:31,833 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-23 13:31:31,833 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-23 13:31:31,835 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-23 13:31:34,250 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 |
| 2025-11-23 13:31:34,251 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-23 13:31:34,251 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-23 13:31:34,252 - __main__ - INFO - chrF: 1.12 |
| 2025-11-23 13:31:34,252 - __main__ - INFO - COMET-QE: 0.0001 |
| 2025-11-23 13:31:34,252 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 13:31:34,263 - __main__ - INFO - 💾 Final results saved to models/finetuned-sw-en-phase2/test_results.json |
| 2025-11-23 13:31:34,270 - __main__ - INFO - ✅ Verified test_results.json with 14 metrics |
| 2025-11-23 13:31:34,272 - __main__ - INFO - |
| 📊 Logging all metrics to TensorBoard... |
| 2025-11-23 13:31:34,531 - __main__ - INFO - ✅ All metrics logged to TensorBoard |
| 2025-11-23 13:31:34,533 - __main__ - INFO - Baseline metrics: 13 |
| 2025-11-23 13:31:34,535 - __main__ - INFO - Final test metrics: 10 |
| 2025-11-23 13:31:34,536 - __main__ - INFO - Final domain metrics: 4 |
| 2025-11-23 13:31:34,537 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-23 13:31:34,537 - __main__ - INFO - 📊 BASELINE vs FINE-TUNED Comparison |
| 2025-11-23 13:31:34,538 - __main__ - INFO - ====================================================================== |
| 2025-11-23 13:31:34,538 - __main__ - INFO - |
| 📈 Test Set Metrics: |
| 2025-11-23 13:31:34,539 - __main__ - INFO - Metric Baseline Fine-tuned Improvement |
| 2025-11-23 13:31:34,540 - __main__ - INFO - ------------------------------------------------------------ |
| 2025-11-23 13:31:34,540 - __main__ - INFO - eval_bleu 0.0000 0.1327 N/A |
| 2025-11-23 13:31:34,542 - __main__ - INFO - eval_chrf 0.0000 32.5863 N/A |
| 2025-11-23 13:31:34,542 - __main__ - INFO - eval_comet 0.0000 0.1811 N/A |
| 2025-11-23 13:31:34,543 - __main__ - INFO - |
| 📈 Domain Metrics (Real Call Transcriptions): |
| 2025-11-23 13:31:34,543 - __main__ - INFO - Metric Baseline Fine-tuned Improvement |
| 2025-11-23 13:31:34,544 - __main__ - INFO - ------------------------------------------------------------ |
| 2025-11-23 13:31:34,545 - __main__ - INFO - domain_bleu 0.0000 0.0000 -99.2% |
| 2025-11-23 13:31:34,545 - __main__ - INFO - domain_chrf 1.2250 1.1158 -8.9% |
| 2025-11-23 13:31:34,546 - __main__ - INFO - domain_comet 0.0001 0.0001 +1.4% |
| 2025-11-23 13:31:34,546 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-23 13:31:34,986 - __main__ - INFO - ✅ Logged 14 final metrics to MLflow |
| 2025-11-23 13:31:34,988 - __main__ - INFO - Visit MLflow: https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-23 13:31:34,989 - __main__ - INFO - 📊 Captured 1601 training log entries |
| 2025-11-23 13:31:35,055 - __main__ - INFO - 💾 Saved comprehensive metrics to models/finetuned-sw-en-phase2/training_metrics.json |
| 2025-11-23 13:31:35,065 - __main__ - INFO - 💾 Saved training config to models/finetuned-sw-en-phase2/training_config_used.json |
|
|
