| 2025-11-26 12:52:01,389 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-26 12:52:01,419 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-26 12:52:01,420 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 12:52:01,421 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-26 12:52:01,424 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-26 12:52:01,425 - __main__ - ERROR - |
| ====================================================================== |
| 2025-11-26 12:52:01,426 - __main__ - ERROR - ❌ HF_TOKEN NOT FOUND! |
| 2025-11-26 12:52:01,426 - __main__ - ERROR - ====================================================================== |
| 2025-11-26 12:52:01,427 - __main__ - ERROR - push_to_hub is enabled but no HuggingFace token is available. |
| 2025-11-26 12:52:01,427 - __main__ - ERROR - |
| To fix this, either: |
| 2025-11-26 12:52:01,428 - __main__ - ERROR - 1. Set HF_TOKEN in your .env file: |
| 2025-11-26 12:52:01,429 - __main__ - ERROR - HF_TOKEN=hf_xxxxx |
| 2025-11-26 12:52:01,430 - __main__ - ERROR - 2. Or export it in your shell: |
| 2025-11-26 12:52:01,430 - __main__ - ERROR - export HF_TOKEN=hf_xxxxx |
| 2025-11-26 12:52:01,431 - __main__ - ERROR - 3. Or disable push_to_hub in your config: |
| 2025-11-26 12:52:01,432 - __main__ - ERROR - "push_to_hub": false |
| 2025-11-26 12:52:01,433 - __main__ - ERROR - ====================================================================== |
|
|
| 2025-11-26 13:01:56,894 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-26 13:01:56,929 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-26 13:01:56,930 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 13:01:56,931 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-26 13:01:56,935 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-26 13:01:56,941 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-26 13:01:56,943 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 13:01:56,944 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-26 13:01:56,946 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-26 13:01:56,947 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-26 13:01:56,948 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... |
| 2025-11-26 13:01:56,949 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... |
| 2025-11-26 13:01:56,960 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-26 13:01:56,961 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 13:01:57,844 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-26 13:01:59,498 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_130158 |
| 2025-11-26 13:01:59,514 - __main__ - INFO - Run ID: 943aef604047491e932c794709ae64e0 |
| 2025-11-26 13:01:59,515 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 13:02:00,428 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-26 13:02:00,432 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-26 13:02:00,451 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-26 13:02:00,451 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-26 13:02:00,452 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 13:02:09,010 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-26 13:02:09,022 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-26 13:02:09,447 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-26 13:02:09,721 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-26 13:02:09,722 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-26 13:02:13,017 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-26 13:02:16,033 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:18,985 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:21,608 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:24,027 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:26,435 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:28,868 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:31,759 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:34,502 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:36,901 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:39,333 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:41,749 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:44,399 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:46,918 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:49,320 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:51,725 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:54,140 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:56,554 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-26 13:02:59,008 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-26 13:03:01,447 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-26 13:03:03,861 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-26 13:03:06,242 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-26 13:03:08,815 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-26 13:03:11,442 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-26 13:03:14,195 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-26 13:03:16,608 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-26 13:03:16,848 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-26 13:03:16,881 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-26 13:03:18,987 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-26 13:03:21,938 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:24,462 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:26,865 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:29,237 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:31,666 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:34,113 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:36,520 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:38,967 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:41,371 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:43,793 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-26 13:03:45,416 - __main__ - WARNING - |
| ⚠️ Training interrupted by user |
| 2025-11-26 13:03:47,340 - __main__ - INFO - ✅ MLflow run ended: 943aef604047491e932c794709ae64e0 |
| 2025-11-26 13:04:46,954 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-26 13:04:46,964 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-26 13:04:46,965 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 13:04:46,966 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-26 13:04:46,970 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-26 13:04:46,971 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-26 13:04:46,972 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 13:04:46,972 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-26 13:04:46,973 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-26 13:04:46,975 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-26 13:04:46,976 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... |
| 2025-11-26 13:04:46,977 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... |
| 2025-11-26 13:04:46,995 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-26 13:04:46,997 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 13:04:47,774 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-26 13:04:48,874 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_130448 |
| 2025-11-26 13:04:48,876 - __main__ - INFO - Run ID: e1050fe493a94824b818f08f28d430f0 |
| 2025-11-26 13:04:48,877 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 13:04:49,354 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-26 13:04:49,358 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-26 13:04:49,365 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-26 13:04:49,366 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-26 13:04:49,367 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 13:04:51,598 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-26 13:04:51,615 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-26 13:04:52,040 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-26 13:04:52,328 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-26 13:04:52,329 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-26 13:04:52,521 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-26 13:04:54,897 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-26 13:04:57,451 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:00,232 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:02,639 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:05,043 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:07,478 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:09,861 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:12,452 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:14,945 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:17,372 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:19,794 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:22,553 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:24,981 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:27,398 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:29,842 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:32,249 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:34,653 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:37,067 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:39,861 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:42,277 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:44,672 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:47,628 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:50,193 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:52,618 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:55,055 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-26 13:05:55,296 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-26 13:05:55,358 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-26 13:05:55,568 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-26 13:05:58,047 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:00,483 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:02,890 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:05,304 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:08,205 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:10,666 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:13,087 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:15,493 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:17,903 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:20,320 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:22,778 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:25,558 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:28,067 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:30,508 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:32,949 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:35,467 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 |
| 2025-11-26 13:06:37,702 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) |
| 2025-11-26 13:06:37,760 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 |
| 2025-11-26 13:06:37,762 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-26 13:06:37,763 - __main__ - INFO - Sample 1: |
| 2025-11-26 13:06:37,766 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... |
| 2025-11-26 13:06:37,767 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... |
| 2025-11-26 13:06:37,767 - __main__ - INFO - Sample 2: |
| 2025-11-26 13:06:37,768 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... |
| 2025-11-26 13:06:37,771 - __main__ - INFO - Target (6 words): Job had said no such thing.... |
| 2025-11-26 13:06:37,772 - __main__ - INFO - Sample 3: |
| 2025-11-26 13:06:37,773 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... |
| 2025-11-26 13:06:37,775 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... |
| 2025-11-26 13:06:43,119 - __main__ - INFO - Filtering statistics: {'kept': 4202243} |
| 2025-11-26 13:06:43,125 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 |
| 2025-11-26 13:06:47,486 - __main__ - INFO - Removed 518 duplicates |
| 2025-11-26 13:06:47,701 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 |
| 2025-11-26 13:06:54,433 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 13:06:54,445 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-26 13:06:54,448 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 13:06:54,451 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 13:06:54,452 - __main__ - INFO - Test: 210,087 samples |
| 2025-11-26 13:06:54,453 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 13:06:54,894 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-26 13:10:05,211 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-26 13:10:05,223 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 13:10:05,225 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 13:10:05,227 - __main__ - INFO - Test: 210,087 samples |
| |
| 2025-11-26 13:10:07,926 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: 59e2a5c4) |
| 2025-11-26 13:10:07,934 - __main__ - INFO - Next run will skip tokenization and load from cache |
| 2025-11-26 13:10:07,936 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... |
| 2025-11-26 13:10:34,363 - __main__ - ERROR - |
| ❌ Pipeline failed: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. |
| Traceback (most recent call last): |
| File "/workspace/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1992, in main |
| trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/workspace/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1331, in __init__ |
| self.tokenized_datasets = self.dataset_dict.map( |
| ^^^^^^^^^^^^^^^^^^^^^^ |
| File "/venv/lib/python3.12/site-packages/datasets/dataset_dict.py", line 953, in map |
| dataset_dict[split] = dataset.map( |
| ^^^^^^^^^^^^ |
| File "/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 562, in wrapper |
| out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3332, in map |
| for rank, done, content in iflatmap_unordered( |
| File "/venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 619, in iflatmap_unordered |
| raise RuntimeError( |
| RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. |
| 2025-11-26 13:10:36,392 - __main__ - INFO - ✅ MLflow run ended: e1050fe493a94824b818f08f28d430f0 |
| 2025-11-26 14:17:28,234 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-26 14:17:28,246 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-26 14:17:28,247 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 14:17:28,248 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-26 14:17:28,251 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-26 14:17:28,252 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-26 14:17:28,253 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 14:17:28,255 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-26 14:17:28,256 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-26 14:17:28,257 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-26 14:17:28,258 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... |
| 2025-11-26 14:17:28,260 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... |
| 2025-11-26 14:17:28,277 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-26 14:17:28,278 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 14:17:29,065 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-26 14:17:30,255 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_141729 |
| 2025-11-26 14:17:30,256 - __main__ - INFO - Run ID: 7185822b56ad49388b2566fd03e2a229 |
| 2025-11-26 14:17:30,261 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 14:17:30,730 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-26 14:17:30,734 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-26 14:17:30,753 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-26 14:17:30,754 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-26 14:17:30,755 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 14:17:35,248 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-26 14:17:35,250 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-26 14:17:35,722 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-26 14:17:36,035 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-26 14:17:36,037 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-26 14:17:39,086 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-26 14:17:41,999 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-26 14:17:44,556 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-26 14:17:47,056 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-26 14:17:49,518 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-26 14:17:51,962 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-26 14:17:54,480 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-26 14:17:56,916 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-26 14:17:59,364 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:01,847 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:04,352 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:06,881 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:09,358 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:11,829 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:14,330 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:16,847 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:19,295 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:21,734 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:24,181 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:26,628 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:29,053 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:31,486 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:33,936 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:36,408 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:38,861 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:41,313 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-26 14:18:41,558 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-26 14:18:41,590 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-26 14:18:42,885 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-26 14:18:45,399 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-26 14:18:47,870 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-26 14:18:50,286 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-26 14:18:52,733 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-26 14:18:55,188 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-26 14:18:57,622 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:00,054 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:02,462 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:04,871 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:07,288 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:09,700 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:12,140 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:14,574 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:16,993 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:19,449 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:21,849 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 |
| 2025-11-26 14:19:24,108 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) |
| 2025-11-26 14:19:24,165 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 |
| 2025-11-26 14:19:24,166 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-26 14:19:24,167 - __main__ - INFO - Sample 1: |
| 2025-11-26 14:19:24,168 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... |
| 2025-11-26 14:19:24,169 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... |
| 2025-11-26 14:19:24,170 - __main__ - INFO - Sample 2: |
| 2025-11-26 14:19:24,171 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... |
| 2025-11-26 14:19:24,172 - __main__ - INFO - Target (6 words): Job had said no such thing.... |
| 2025-11-26 14:19:24,173 - __main__ - INFO - Sample 3: |
| 2025-11-26 14:19:24,174 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... |
| 2025-11-26 14:19:24,175 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... |
| 2025-11-26 14:19:29,479 - __main__ - INFO - Filtering statistics: {'kept': 4202243} |
| 2025-11-26 14:19:29,481 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 |
| 2025-11-26 14:19:34,042 - __main__ - INFO - Removed 518 duplicates |
| 2025-11-26 14:19:34,313 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 |
| 2025-11-26 14:19:41,590 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 14:19:41,592 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-26 14:19:41,593 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 14:19:41,594 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 14:19:41,595 - __main__ - INFO - Test: 210,087 samples |
| 2025-11-26 14:19:41,596 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 14:19:42,055 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-26 14:22:54,655 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-26 14:22:54,658 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 14:22:54,660 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 14:22:54,661 - __main__ - INFO - Test: 210,087 samples |
| |
| 2025-11-26 14:22:57,099 - __main__ - INFO - 📝 Will cache tokenized datasets (hash: 59e2a5c4) |
| 2025-11-26 14:22:57,117 - __main__ - INFO - Next run will skip tokenization and load from cache |
| 2025-11-26 14:22:57,120 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... |
| 2025-11-26 14:23:22,483 - __main__ - ERROR - |
| ❌ Pipeline failed: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. |
| Traceback (most recent call last): |
| File "/workspace/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1992, in main |
| trainer_manager = TranslationTrainer(config, model, tokenizer, dataset_dict) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/workspace/openchs_rnd/tasks/translation/swahili/script/translation.py", line 1331, in __init__ |
| self.tokenized_datasets = self.dataset_dict.map( |
| ^^^^^^^^^^^^^^^^^^^^^^ |
| File "/venv/lib/python3.12/site-packages/datasets/dataset_dict.py", line 953, in map |
| dataset_dict[split] = dataset.map( |
| ^^^^^^^^^^^^ |
| File "/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 562, in wrapper |
| out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3332, in map |
| for rank, done, content in iflatmap_unordered( |
| File "/venv/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 619, in iflatmap_unordered |
| raise RuntimeError( |
| RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing. |
| 2025-11-26 14:23:24,870 - __main__ - INFO - ✅ MLflow run ended: 7185822b56ad49388b2566fd03e2a229 |
| 2025-11-26 14:52:14,188 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-26 14:52:14,224 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-26 14:52:14,226 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 14:52:14,227 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-26 14:52:14,232 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-26 14:52:14,233 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-26 14:52:14,234 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 14:52:14,235 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-26 14:52:14,236 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-26 14:52:14,237 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-26 14:52:14,238 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... |
| 2025-11-26 14:52:14,239 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... |
| 2025-11-26 14:52:14,261 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-26 14:52:14,263 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 14:52:15,107 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-26 14:52:16,508 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_145215 |
| 2025-11-26 14:52:16,516 - __main__ - INFO - Run ID: 56240a85d2814079a1e568351bd90cef |
| 2025-11-26 14:52:16,519 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 14:52:16,969 - __main__ - INFO - ✅ Logged 75 config parameters to MLflow |
| 2025-11-26 14:52:16,972 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-26 14:52:17,115 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-26 14:52:17,117 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-26 14:52:17,118 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 14:52:25,971 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-26 14:52:25,973 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-26 14:52:26,457 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-26 14:52:26,739 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-26 14:52:26,740 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-26 14:52:37,140 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-26 14:52:40,247 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-26 14:52:43,241 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-26 14:52:45,871 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-26 14:52:48,447 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-26 14:52:51,105 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-26 14:52:53,747 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-26 14:52:56,348 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-26 14:52:58,964 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:01,618 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:04,186 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:06,761 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:09,408 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:12,072 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:14,680 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:17,357 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:20,027 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:22,667 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:25,362 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:28,018 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:30,678 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:33,290 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:36,230 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:39,427 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:41,946 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:44,471 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-26 14:53:44,716 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-26 14:53:44,752 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-26 14:53:53,135 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-26 14:53:56,107 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-26 14:53:59,022 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:01,537 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:03,970 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:06,435 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:08,903 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:11,364 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:13,833 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:16,294 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:18,765 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:21,232 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:23,706 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:26,181 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:28,627 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:31,077 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:33,538 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 |
| 2025-11-26 14:54:35,808 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) |
| 2025-11-26 14:54:35,867 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 |
| 2025-11-26 14:54:35,868 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-26 14:54:35,871 - __main__ - INFO - Sample 1: |
| 2025-11-26 14:54:35,873 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... |
| 2025-11-26 14:54:35,874 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... |
| 2025-11-26 14:54:35,875 - __main__ - INFO - Sample 2: |
| 2025-11-26 14:54:35,875 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... |
| 2025-11-26 14:54:35,876 - __main__ - INFO - Target (6 words): Job had said no such thing.... |
| 2025-11-26 14:54:35,877 - __main__ - INFO - Sample 3: |
| 2025-11-26 14:54:35,878 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... |
| 2025-11-26 14:54:35,879 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... |
| 2025-11-26 14:54:41,232 - __main__ - INFO - Filtering statistics: {'kept': 4202243} |
| 2025-11-26 14:54:41,233 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 |
| 2025-11-26 14:54:45,682 - __main__ - INFO - Removed 518 duplicates |
| 2025-11-26 14:54:45,894 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 |
| 2025-11-26 14:54:53,322 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 14:54:53,325 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-26 14:54:53,326 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 14:54:53,326 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 14:54:53,327 - __main__ - INFO - Test: 210,087 samples |
| 2025-11-26 14:54:53,328 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-26 14:54:53,835 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-26 14:58:05,583 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-26 14:58:05,586 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 14:58:05,588 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 14:58:05,589 - __main__ - INFO - Test: 210,087 samples |
|
|
| 2025-11-26 14:58:07,067 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) |
| 2025-11-26 14:58:07,068 - __main__ - INFO - Skipping tokenization - this saves significant time! |
| 2025-11-26 14:58:07,069 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... |
| 2025-11-26 14:58:42,055 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-26 14:58:42,057 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-26 14:58:42,159 - comet.models.download_utils - INFO - Downloading wmt20-comet-qe-da.tar.gz |
| 2025-11-26 14:59:21,457 - comet.models.download_utils - INFO - Extracting /root/.cache/torch/unbabel_comet/wmt20-comet-qe-da.tar.gz |
| 2025-11-26 14:59:39,749 - comet.models.download_utils - INFO - Extracted /root/.cache/torch/unbabel_comet/wmt20-comet-qe-da.tar.gz |
| 2025-11-26 14:59:41,205 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-26 14:59:49,994 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-26 14:59:50,547 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-26 14:59:50,549 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-26 14:59:50,597 - __main__ - INFO - ✅ Trainer initialized |
| 2025-11-26 14:59:50,598 - __main__ - INFO - |
| 🔬 STEP 1: Baseline Evaluation |
| 2025-11-26 14:59:51,327 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 |
| 2025-11-26 14:59:51,328 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 14:59:51,330 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... |
| 2025-11-26 14:59:51,331 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:00:31,842 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1216 |
| 2025-11-26 15:00:31,844 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-26 15:00:31,875 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... |
| 2025-11-26 15:00:31,877 - __main__ - INFO - Batch 1/16... |
| 2025-11-26 15:00:33,243 - __main__ - INFO - Batch 5/16... |
| 2025-11-26 15:00:34,320 - __main__ - INFO - Batch 10/16... |
| 2025-11-26 15:00:35,558 - __main__ - INFO - Batch 15/16... |
| 2025-11-26 15:00:35,946 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-26 15:00:35,958 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-26 15:00:35,967 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-26 15:00:35,967 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-26 15:00:35,968 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision |
| 2025-11-26 15:00:35,970 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-26 15:00:39,691 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 |
| 2025-11-26 15:00:39,692 - __main__ - INFO - |
| ✅ Baseline Results (Test Set): |
| 2025-11-26 15:00:39,692 - __main__ - INFO - BLEU: 0.1216 |
| 2025-11-26 15:00:39,693 - __main__ - INFO - chrF: 32.36 |
| 2025-11-26 15:00:39,693 - __main__ - INFO - COMET: 0.1977 |
| 2025-11-26 15:00:39,695 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-26 15:00:39,695 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:00:39,696 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-26 15:00:39,696 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:00:39,697 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-26 15:00:39,698 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-26 15:00:40,792 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-26 15:00:40,878 - __main__ - INFO - ✅ Domain chrF: 1.22 |
| 2025-11-26 15:00:40,879 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-26 15:00:40,886 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-26 15:00:40,894 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-26 15:00:40,894 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-26 15:00:40,896 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-26 15:00:43,200 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 |
| 2025-11-26 15:00:43,213 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-26 15:00:43,214 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-26 15:00:43,215 - __main__ - INFO - chrF: 1.22 |
| 2025-11-26 15:00:43,216 - __main__ - INFO - COMET-QE: 0.0001 |
| 2025-11-26 15:00:43,217 - __main__ - INFO - ====================================================================== |
|
|
| 2025-11-26 15:00:43,831 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 |
| 2025-11-26 15:00:43,847 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json |
| 2025-11-26 15:00:43,858 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics |
| 2025-11-26 15:00:43,860 - __main__ - INFO - |
| 🚀 STEP 2: Fine-tuning |
| 2025-11-26 15:00:43,861 - __main__ - INFO - Metrics are being logged to MLflow in real-time |
| 2025-11-26 15:00:43,863 - __main__ - INFO - MLflow URL: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 15:00:43,865 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs |
| 2025-11-26 15:00:43,866 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') |
| 2025-11-26 15:00:43,868 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 15:00:43,869 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:00:43,870 - __main__ - INFO - 🚀 Starting training... |
| 2025-11-26 15:00:43,871 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:00:45,126 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) |
| 2025-11-26 15:01:06,238 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.1%) |
| 2025-11-26 15:01:26,944 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.2%) |
| 2025-11-26 15:01:48,288 - __main__ - INFO - 📈 Epoch progress: 0.02/6 (0.4%) |
| 2025-11-26 15:02:09,155 - __main__ - INFO - 📈 Epoch progress: 0.03/6 (0.5%) |
| 2025-11-26 15:02:29,947 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.6%) |
| 2025-11-26 15:02:50,639 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.7%) |
| 2025-11-26 15:03:11,496 - __main__ - INFO - 📈 Epoch progress: 0.05/6 (0.8%) |
| 2025-11-26 15:03:32,002 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.0%) |
| 2025-11-26 15:03:52,078 - urllib3.connectionpool - WARNING - Retrying (Retry(total=6, connect=7, read=6, redirect=7, status=7)) after connection broken by 'RemoteDisconnected('Remote end closed connection without response')': /api/2.0/mlflow/runs/log-batch |
| 2025-11-26 15:03:52,531 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.1%) |
| 2025-11-26 15:04:13,047 - __main__ - INFO - 📈 Epoch progress: 0.07/6 (1.2%) |
| 2025-11-26 15:19:54,631 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-26 15:19:54,643 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-26 15:19:54,643 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 15:19:54,645 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-26 15:19:54,647 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-26 15:19:54,648 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-26 15:19:54,649 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 15:19:54,651 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-26 15:19:54,651 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-26 15:19:54,652 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-26 15:19:54,652 - __main__ - INFO - 🔍 Checking connectivity to https://food-accepts-blocked-refinance.trycloudflare.com... |
| 2025-11-26 15:19:54,653 - __main__ - INFO - 🔍 Checking connectivity to food-accepts-blocked-refinance.trycloudflare.com:443... |
| 2025-11-26 15:19:54,675 - __main__ - WARNING - ⚠️ DNS resolution failed: [Errno -2] Name or service not known |
| 2025-11-26 15:19:54,676 - __main__ - ERROR - ❌ Cannot reach MLflow server at https://food-accepts-blocked-refinance.trycloudflare.com |
| 2025-11-26 15:19:54,677 - __main__ - ERROR - Will proceed with file logging only (MLflow disabled) |
| 2025-11-26 15:19:54,679 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-26 15:19:54,716 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-26 15:19:54,717 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-26 15:19:54,718 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 15:19:59,423 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-26 15:19:59,425 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-26 15:19:59,839 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-26 15:20:00,106 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-26 15:20:00,116 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-26 15:20:08,129 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-26 15:20:10,899 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:13,823 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:16,634 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:19,094 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:21,446 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:23,821 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:26,130 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:28,448 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:30,819 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:33,225 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:35,647 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:38,038 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:40,449 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:42,846 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:45,232 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:47,601 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:49,975 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:52,373 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:55,083 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:57,461 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-26 15:20:59,815 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-26 15:21:02,179 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-26 15:21:04,556 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-26 15:21:06,954 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-26 15:21:09,332 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-26 15:21:09,569 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-26 15:21:09,608 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-26 15:21:13,615 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-26 15:21:16,561 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:19,432 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:21,830 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:24,197 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:26,610 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:29,005 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:31,439 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:33,860 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:36,301 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:38,899 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:41,759 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:44,223 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:46,651 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:49,074 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:51,521 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:53,990 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 |
| 2025-11-26 15:21:56,241 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) |
| 2025-11-26 15:21:56,290 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 |
| 2025-11-26 15:21:56,291 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-26 15:21:56,292 - __main__ - INFO - Sample 1: |
| 2025-11-26 15:21:56,292 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... |
| 2025-11-26 15:21:56,293 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... |
| 2025-11-26 15:21:56,293 - __main__ - INFO - Sample 2: |
| 2025-11-26 15:21:56,294 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... |
| 2025-11-26 15:21:56,295 - __main__ - INFO - Target (6 words): Job had said no such thing.... |
| 2025-11-26 15:21:56,295 - __main__ - INFO - Sample 3: |
| 2025-11-26 15:21:56,296 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... |
| 2025-11-26 15:21:56,296 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... |
| 2025-11-26 15:22:01,627 - __main__ - INFO - Filtering statistics: {'kept': 4202243} |
| 2025-11-26 15:22:01,628 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 |
| 2025-11-26 15:22:06,238 - __main__ - INFO - Removed 518 duplicates |
| 2025-11-26 15:22:06,460 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 |
| 2025-11-26 15:22:13,232 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:22:13,233 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-26 15:22:13,233 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 15:22:13,234 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 15:22:13,235 - __main__ - INFO - Test: 210,087 samples |
| 2025-11-26 15:22:13,235 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 15:22:13,701 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-26 15:25:20,487 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-26 15:25:20,490 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 15:25:20,490 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 15:25:20,491 - __main__ - INFO - Test: 210,087 samples |
| |
| 2025-11-26 15:25:21,869 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) |
| 2025-11-26 15:25:21,870 - __main__ - INFO - Skipping tokenization - this saves significant time! |
| 2025-11-26 15:25:21,871 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... |
| 2025-11-26 15:26:03,304 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-26 15:26:03,318 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-26 15:26:03,452 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-26 15:26:04,582 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-26 15:26:10,765 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-26 15:26:11,304 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-26 15:26:11,320 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-26 15:26:11,352 - __main__ - INFO - 📊 Limiting evaluation dataset from 420225 to 2000 samples |
| 2025-11-26 15:26:11,366 - __main__ - INFO - ✅ Trainer initialized |
| 2025-11-26 15:26:11,367 - __main__ - INFO - |
| 🔬 STEP 1: Baseline Evaluation |
| 2025-11-26 15:26:11,367 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:26:11,368 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... |
| 2025-11-26 15:26:11,368 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:26:43,143 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1215 |
| 2025-11-26 15:26:43,145 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-26 15:26:43,182 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... |
| 2025-11-26 15:26:43,183 - __main__ - INFO - Batch 1/16... |
| 2025-11-26 15:26:44,523 - __main__ - INFO - Batch 5/16... |
| 2025-11-26 15:26:45,642 - __main__ - INFO - Batch 10/16... |
| 2025-11-26 15:26:46,850 - __main__ - INFO - Batch 15/16... |
| 2025-11-26 15:26:47,234 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-26 15:26:47,246 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-26 15:26:47,255 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-26 15:26:47,255 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-26 15:26:47,256 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision |
| 2025-11-26 15:26:47,258 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-26 15:26:50,282 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 |
| 2025-11-26 15:26:50,283 - __main__ - INFO - |
| ✅ Baseline Results (Test Set): |
| 2025-11-26 15:26:50,285 - __main__ - INFO - BLEU: 0.1215 |
| 2025-11-26 15:26:50,286 - __main__ - INFO - chrF: 32.36 |
| 2025-11-26 15:26:50,286 - __main__ - INFO - COMET: 0.1977 |
| 2025-11-26 15:26:50,287 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 15:26:50,288 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:26:50,288 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-26 15:26:50,289 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:26:50,290 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-26 15:26:50,291 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-26 15:26:51,371 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-26 15:26:51,431 - __main__ - INFO - ✅ Domain chrF: 1.22 |
| 2025-11-26 15:26:51,432 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-26 15:26:51,438 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-26 15:26:51,447 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-26 15:26:51,449 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-26 15:26:51,452 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-26 15:26:53,552 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 |
| 2025-11-26 15:26:53,553 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-26 15:26:53,554 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-26 15:26:53,555 - __main__ - INFO - chrF: 1.22 |
| 2025-11-26 15:26:53,555 - __main__ - INFO - COMET-QE: 0.0001 |
| 2025-11-26 15:26:53,556 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 15:26:53,565 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json |
| 2025-11-26 15:26:53,570 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics |
| 2025-11-26 15:26:53,570 - __main__ - INFO - |
| 🚀 STEP 2: Fine-tuning |
| 2025-11-26 15:26:53,571 - __main__ - INFO - Metrics are being logged to MLflow in real-time |
| 2025-11-26 15:26:53,571 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs |
| 2025-11-26 15:26:53,572 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') |
| 2025-11-26 15:26:53,572 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 15:26:53,573 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:26:53,573 - __main__ - INFO - 🚀 Starting training... |
| 2025-11-26 15:26:53,574 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:34:05,454 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1169 |
| 2025-11-26 15:34:13,047 - __main__ - INFO - ✅ Repository brendaogutu/mul-sw-en-translation-phase2 ready |
| 2025-11-26 15:34:13,049 - __main__ - INFO - |
| 📤 Pushing checkpoint 2000 to HuggingFace Hub... |
| 2025-11-26 15:34:13,050 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 15:34:33,303 - __main__ - INFO - ✅ Checkpoint 2000 pushed successfully! |
| 2025-11-26 15:34:33,316 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-2000 |
| 2025-11-26 15:38:14,833 - __main__ - INFO - ✅ COMET available for evaluation |
| 2025-11-26 15:38:14,848 - __main__ - INFO - ✅ Configuration loaded from configs/swahili_runpod_phase2.json |
| 2025-11-26 15:38:14,850 - __main__ - INFO - Base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 15:38:14,850 - __main__ - INFO - Model output: /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2 |
| 2025-11-26 15:38:14,855 - __main__ - INFO - 🎲 Random seed: 42 |
| 2025-11-26 15:38:14,856 - __main__ - INFO - ✅ HuggingFace token found: hf_JeMkwQl...vJbK |
| 2025-11-26 15:38:14,857 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 15:38:14,857 - __main__ - INFO - Push strategy: checkpoint |
| 2025-11-26 15:38:14,864 - __main__ - INFO - → Will push after each checkpoint save |
| 2025-11-26 15:38:14,865 - __main__ - INFO - ✅ MLflowManager initialized |
| 2025-11-26 15:38:14,867 - __main__ - INFO - 🔍 Checking connectivity to https://proceedings-graphic-pressed-responded.trycloudflare.com... |
| 2025-11-26 15:38:14,867 - __main__ - INFO - 🔍 Checking connectivity to proceedings-graphic-pressed-responded.trycloudflare.com:443... |
| 2025-11-26 15:38:14,885 - __main__ - INFO - ✅ MLflow server is reachable |
| 2025-11-26 15:38:14,886 - __main__ - INFO - ✅ MLflow tracking URI set: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 15:38:16,109 - __main__ - INFO - ✅ Using experiment: swahili-english-translation |
| 2025-11-26 15:38:17,370 - __main__ - INFO - ✅ MLflow run started: sw-en-translation_20251126_153816 |
| 2025-11-26 15:38:17,371 - __main__ - INFO - Run ID: d9a43848ca0a46218f55909e1f84eaee |
| 2025-11-26 15:38:17,373 - __main__ - INFO - Tracking URI: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 15:38:17,868 - __main__ - INFO - ✅ Logged 76 config parameters to MLflow |
| 2025-11-26 15:38:17,873 - __main__ - INFO - |
| 📥 Loading domain evaluation data... |
| 2025-11-26 15:38:17,898 - __main__ - INFO - ✅ Loaded 5 domain evaluation pairs |
| 2025-11-26 15:38:17,929 - __main__ - INFO - Sample: huduma kwa watoto halo halo habari mzuri, shikamo marahaba unaongea na mshauri A... |
| 2025-11-26 15:38:17,939 - __main__ - INFO - |
| 📥 Loading base model: brendaogutu/mul-sw-en-translation-phase1 |
| 2025-11-26 15:38:22,708 - __main__ - INFO - ✅ Model loaded on cuda:0 |
| 2025-11-26 15:38:22,713 - __main__ - INFO - 🔍 Verifying dataset files... |
| 2025-11-26 15:38:23,112 - __main__ - INFO - ✅ nllb_swahili_3: 2,509,737 lines in file |
| 2025-11-26 15:38:23,381 - __main__ - INFO - ✅ nllb_swahili_4: 1,692,506 lines in file |
| 2025-11-26 15:38:23,382 - __main__ - INFO - 📥 Loading nllb_swahili_3 from /tmp/dataset/NLLB.en-sw-4-cleaned_clean.jsonl |
| 2025-11-26 15:38:36,717 - __main__ - INFO - ✅ Loaded 2,509,737 samples using datasets library (memory-efficient) |
| 2025-11-26 15:38:39,716 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_3 |
| 2025-11-26 15:38:42,275 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_3 |
| 2025-11-26 15:38:44,735 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_3 |
| 2025-11-26 15:38:47,169 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_3 |
| 2025-11-26 15:38:49,607 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_3 |
| 2025-11-26 15:38:52,120 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_3 |
| 2025-11-26 15:38:54,561 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_3 |
| 2025-11-26 15:38:56,986 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_3 |
| 2025-11-26 15:38:59,421 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:01,888 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:04,346 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:06,753 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:09,161 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:11,570 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:13,995 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:16,378 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:18,770 - __main__ - INFO - Processed 1,700,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:21,205 - __main__ - INFO - Processed 1,800,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:23,649 - __main__ - INFO - Processed 1,900,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:26,111 - __main__ - INFO - Processed 2,000,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:28,503 - __main__ - INFO - Processed 2,100,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:30,905 - __main__ - INFO - Processed 2,200,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:33,336 - __main__ - INFO - Processed 2,300,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:35,755 - __main__ - INFO - Processed 2,400,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:38,159 - __main__ - INFO - Processed 2,500,000 samples from nllb_swahili_3 |
| 2025-11-26 15:39:38,393 - __main__ - INFO - Loaded 2,509,737 samples from nllb_swahili_3 (weight: 1.0x) |
| 2025-11-26 15:39:38,430 - __main__ - INFO - 📥 Loading nllb_swahili_4 from /tmp/dataset/NLLB.en-sw-5-cleaned_clean.jsonl |
| 2025-11-26 15:39:41,057 - __main__ - INFO - ✅ Loaded 1,692,506 samples using datasets library (memory-efficient) |
| 2025-11-26 15:39:43,501 - __main__ - INFO - Processed 100,000 samples from nllb_swahili_4 |
| 2025-11-26 15:39:45,923 - __main__ - INFO - Processed 200,000 samples from nllb_swahili_4 |
| 2025-11-26 15:39:48,326 - __main__ - INFO - Processed 300,000 samples from nllb_swahili_4 |
| 2025-11-26 15:39:50,702 - __main__ - INFO - Processed 400,000 samples from nllb_swahili_4 |
| 2025-11-26 15:39:53,128 - __main__ - INFO - Processed 500,000 samples from nllb_swahili_4 |
| 2025-11-26 15:39:55,545 - __main__ - INFO - Processed 600,000 samples from nllb_swahili_4 |
| 2025-11-26 15:39:57,960 - __main__ - INFO - Processed 700,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:00,396 - __main__ - INFO - Processed 800,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:02,809 - __main__ - INFO - Processed 900,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:05,238 - __main__ - INFO - Processed 1,000,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:07,634 - __main__ - INFO - Processed 1,100,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:10,037 - __main__ - INFO - Processed 1,200,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:12,439 - __main__ - INFO - Processed 1,300,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:14,830 - __main__ - INFO - Processed 1,400,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:17,241 - __main__ - INFO - Processed 1,500,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:19,628 - __main__ - INFO - Processed 1,600,000 samples from nllb_swahili_4 |
| 2025-11-26 15:40:21,855 - __main__ - INFO - Loaded 1,692,506 samples from nllb_swahili_4 (weight: 1.0x) |
| 2025-11-26 15:40:21,899 - __main__ - INFO - 📊 Total samples loaded: 4,202,243 |
| 2025-11-26 15:40:21,914 - __main__ - INFO - |
| 📝 Sample data check: |
| 2025-11-26 15:40:21,916 - __main__ - INFO - Sample 1: |
| 2025-11-26 15:40:21,917 - __main__ - INFO - Source (11 words): Ni kitabu cha watoto, lakini umuhimu wake ni wa jamii nzima.... |
| 2025-11-26 15:40:21,917 - __main__ - INFO - Target (12 words): It is a children's book, but it is also a grown-up book.... |
| 2025-11-26 15:40:21,918 - __main__ - INFO - Sample 2: |
| 2025-11-26 15:40:21,919 - __main__ - INFO - Source (5 words): Ayubu anaamini kuwa haitakuwa hivyo.... |
| 2025-11-26 15:40:21,920 - __main__ - INFO - Target (6 words): Job had said no such thing.... |
| 2025-11-26 15:40:21,920 - __main__ - INFO - Sample 3: |
| 2025-11-26 15:40:21,921 - __main__ - INFO - Source (16 words): "Kwa hiyo, niseme kwamba chochote ambacho kinaongelewa kwenye mitandao ya kijami... |
| 2025-11-26 15:40:21,921 - __main__ - INFO - Target (15 words): Later I realized the fact that everything which is shown on TV is not true.... |
| 2025-11-26 15:40:27,246 - __main__ - INFO - Filtering statistics: {'kept': 4202243} |
| 2025-11-26 15:40:27,250 - __main__ - INFO - 📊 Samples after filtering: 4,202,243 |
| 2025-11-26 15:40:31,728 - __main__ - INFO - Removed 518 duplicates |
| 2025-11-26 15:40:31,942 - __main__ - INFO - 📊 Samples after deduplication: 4,201,725 |
| 2025-11-26 15:40:38,598 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:40:38,598 - __main__ - INFO - ✅ Dataset splits: |
| 2025-11-26 15:40:38,615 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 15:40:38,616 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 15:40:38,617 - __main__ - INFO - Test: 210,087 samples |
| 2025-11-26 15:40:38,618 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 15:40:39,055 - __main__ - INFO - |
| 🔍 Validating dataset before training... |
| 2025-11-26 15:43:48,131 - __main__ - INFO - ✅ Dataset validation passed |
| 2025-11-26 15:43:48,134 - __main__ - INFO - Train: 3,571,465 samples |
| 2025-11-26 15:43:48,135 - __main__ - INFO - Validation: 420,173 samples |
| 2025-11-26 15:43:48,135 - __main__ - INFO - Test: 210,087 samples |
| |
| 2025-11-26 15:43:49,920 - __main__ - INFO - ✅ Found cached tokenized datasets (hash: 59e2a5c4) |
| 2025-11-26 15:43:49,921 - __main__ - INFO - Skipping tokenization - this saves significant time! |
| 2025-11-26 15:43:49,922 - __main__ - INFO - Tokenizing datasets (batch_size=1000, num_proc=1)... |
| 2025-11-26 15:44:27,150 - __main__ - INFO - ✅ Tokenization complete! |
| 2025-11-26 15:44:27,152 - __main__ - INFO - 🔧 Initializing COMET-QE model... |
| 2025-11-26 15:44:27,269 - comet.models.download_utils - INFO - wmt20-comet-qe-da is already in cache. |
| 2025-11-26 15:44:28,434 - pytorch_lightning.utilities.migration.utils - INFO - Lightning automatically upgraded your loaded checkpoint from v1.3.5 to v2.5.6. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../../../../../root/.cache/torch/unbabel_comet/wmt20-comet-qe-da/checkpoints/model.ckpt` |
| 2025-11-26 15:44:34,787 - comet.models.base - INFO - Encoder model frozen. |
| 2025-11-26 15:44:35,315 - __main__ - INFO - ✅ COMET-QE model loaded successfully |
| 2025-11-26 15:44:35,316 - __main__ - INFO - ✅ TensorBoard enabled - logs will be saved to logs/ |
| 2025-11-26 15:44:35,351 - __main__ - INFO - 📊 Limiting evaluation dataset from 420225 to 2000 samples |
| 2025-11-26 15:44:35,365 - __main__ - INFO - ✅ Trainer initialized |
| 2025-11-26 15:44:35,366 - __main__ - INFO - |
| 🔬 STEP 1: Baseline Evaluation |
| 2025-11-26 15:44:35,931 - __main__ - INFO - ✅ MLflow: Baseline evaluation starting - status_code=0.5 |
| 2025-11-26 15:44:35,932 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:44:35,933 - __main__ - INFO - 📊 Evaluating BASELINE (pre-trained) model on test set... |
| 2025-11-26 15:44:35,933 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:45:07,851 - __main__ - INFO - 📊 Evaluation 1: BLEU = 0.1215 |
| 2025-11-26 15:45:07,854 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-26 15:45:07,891 - __main__ - INFO - Computing predictions for COMET (500 samples, batch_size=32)... |
| 2025-11-26 15:45:07,892 - __main__ - INFO - Batch 1/16... |
| 2025-11-26 15:45:09,406 - __main__ - INFO - Batch 5/16... |
| 2025-11-26 15:45:10,628 - __main__ - INFO - Batch 10/16... |
| 2025-11-26 15:45:12,010 - __main__ - INFO - Batch 15/16... |
| 2025-11-26 15:45:12,404 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-26 15:45:12,443 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-26 15:45:12,451 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-26 15:45:12,452 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-26 15:45:12,453 - pytorch_lightning.utilities.rank_zero - INFO - You are using a CUDA device ('NVIDIA A40') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision |
| 2025-11-26 15:45:12,456 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-26 15:45:16,167 - __main__ - INFO - ✅ COMET-QE: 0.1977 ± 0.2784 |
| 2025-11-26 15:45:16,168 - __main__ - INFO - |
| ✅ Baseline Results (Test Set): |
| 2025-11-26 15:45:16,169 - __main__ - INFO - BLEU: 0.1215 |
| 2025-11-26 15:45:16,170 - __main__ - INFO - chrF: 32.36 |
| 2025-11-26 15:45:16,170 - __main__ - INFO - COMET: 0.1977 |
| 2025-11-26 15:45:16,170 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 15:45:16,171 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:45:16,171 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-26 15:45:16,172 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:45:16,172 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-26 15:45:16,173 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-26 15:45:17,286 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-26 15:45:17,373 - __main__ - INFO - ✅ Domain chrF: 1.22 |
| 2025-11-26 15:45:17,374 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-26 15:45:17,380 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-26 15:45:17,389 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-26 15:45:17,390 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-26 15:45:17,392 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-26 15:45:19,457 - __main__ - INFO - ✅ Domain COMET-QE: 0.0001 |
| 2025-11-26 15:45:19,458 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-26 15:45:19,460 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-26 15:45:19,461 - __main__ - INFO - chrF: 1.22 |
| 2025-11-26 15:45:19,462 - __main__ - INFO - COMET-QE: 0.0001 |
| 2025-11-26 15:45:19,463 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 15:45:19,966 - __main__ - INFO - ✅ Logged 13 baseline metrics to MLflow at step 0 |
| 2025-11-26 15:45:19,978 - __main__ - INFO - 💾 Baseline results saved to models/finetuned-sw-en-phase2/baseline_results.json |
| 2025-11-26 15:45:19,988 - __main__ - INFO - ✅ Verified baseline_results.json with 13 metrics |
| 2025-11-26 15:45:19,989 - __main__ - INFO - |
| 🚀 STEP 2: Fine-tuning |
| 2025-11-26 15:45:19,990 - __main__ - INFO - Metrics are being logged to MLflow in real-time |
| 2025-11-26 15:45:19,991 - __main__ - INFO - MLflow URL: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 15:45:19,992 - __main__ - INFO - TensorBoard: tensorboard --logdir /workspace/openchs_rnd/tasks/translation/swahili/models/finetuned-sw-en-phase2/logs |
| 2025-11-26 15:45:19,994 - __main__ - INFO - ✅ HuggingFace Hub callback enabled (push_strategy='checkpoint') |
| 2025-11-26 15:45:19,995 - __main__ - INFO - Checkpoints will be pushed to: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 15:45:19,996 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 15:45:19,997 - __main__ - INFO - 🚀 Starting training... |
| 2025-11-26 15:45:19,998 - __main__ - INFO - ====================================================================== |
| 2025-11-26 15:45:21,185 - __main__ - INFO - ✅ MLflow: Training started - status_code=1.0 (TRAINING) |
| 2025-11-26 15:45:42,465 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.1%) |
| 2025-11-26 15:46:03,034 - __main__ - INFO - 📈 Epoch progress: 0.01/6 (0.2%) |
| 2025-11-26 15:46:24,110 - __main__ - INFO - 📈 Epoch progress: 0.02/6 (0.4%) |
| 2025-11-26 15:46:45,084 - __main__ - INFO - 📈 Epoch progress: 0.03/6 (0.5%) |
| 2025-11-26 15:47:05,886 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.6%) |
| 2025-11-26 15:47:26,381 - __main__ - INFO - 📈 Epoch progress: 0.04/6 (0.7%) |
| 2025-11-26 15:47:47,439 - __main__ - INFO - 📈 Epoch progress: 0.05/6 (0.8%) |
| 2025-11-26 15:48:08,096 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.0%) |
| 2025-11-26 15:48:28,758 - __main__ - INFO - 📈 Epoch progress: 0.06/6 (1.1%) |
| 2025-11-26 15:48:50,137 - __main__ - INFO - 📈 Epoch progress: 0.07/6 (1.2%) |
| 2025-11-26 15:49:11,147 - __main__ - INFO - 📈 Epoch progress: 0.08/6 (1.3%) |
| 2025-11-26 15:49:31,904 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.4%) |
| 2025-11-26 15:49:52,850 - __main__ - INFO - 📈 Epoch progress: 0.09/6 (1.6%) |
| 2025-11-26 15:50:13,654 - __main__ - INFO - 📈 Epoch progress: 0.10/6 (1.7%) |
| 2025-11-26 15:50:34,024 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.8%) |
| 2025-11-26 15:50:54,355 - __main__ - INFO - 📈 Epoch progress: 0.11/6 (1.9%) |
| 2025-11-26 15:51:15,161 - __main__ - INFO - 📈 Epoch progress: 0.12/6 (2.0%) |
| 2025-11-26 15:51:36,131 - __main__ - INFO - 📈 Epoch progress: 0.13/6 (2.2%) |
| 2025-11-26 15:51:56,801 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.3%) |
| 2025-11-26 15:52:17,573 - __main__ - INFO - 📈 Epoch progress: 0.14/6 (2.4%) |
| 2025-11-26 15:53:04,601 - __main__ - INFO - 📊 Evaluation 2: BLEU = 0.1169 |
| 2025-11-26 15:53:05,031 - __main__ - INFO - 📊 Evaluation at step 2000 |
| 2025-11-26 15:53:09,974 - __main__ - INFO - ✅ Repository brendaogutu/mul-sw-en-translation-phase2 ready |
| 2025-11-26 15:53:09,975 - __main__ - INFO - |
| 📤 Pushing checkpoint 2000 to HuggingFace Hub... |
| 2025-11-26 15:53:09,976 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 15:53:17,857 - __main__ - INFO - ✅ Checkpoint 2000 pushed successfully! |
| 2025-11-26 15:53:17,859 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-2000 |
| 2025-11-26 15:53:38,219 - __main__ - INFO - 📈 Epoch progress: 0.15/6 (2.5%) |
| 2025-11-26 15:53:59,290 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.6%) |
| 2025-11-26 15:54:20,049 - __main__ - INFO - 📈 Epoch progress: 0.16/6 (2.7%) |
| 2025-11-26 15:54:40,579 - __main__ - INFO - 📈 Epoch progress: 0.17/6 (2.9%) |
| 2025-11-26 15:55:01,123 - __main__ - INFO - 📈 Epoch progress: 0.18/6 (3.0%) |
| 2025-11-26 15:55:21,745 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.1%) |
| 2025-11-26 15:55:41,969 - __main__ - INFO - 📈 Epoch progress: 0.19/6 (3.2%) |
| 2025-11-26 15:56:03,309 - __main__ - INFO - 📈 Epoch progress: 0.20/6 (3.3%) |
| 2025-11-26 15:56:23,794 - __main__ - INFO - 📈 Epoch progress: 0.21/6 (3.5%) |
| 2025-11-26 15:56:44,598 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.6%) |
| 2025-11-26 15:57:05,030 - __main__ - INFO - 📈 Epoch progress: 0.22/6 (3.7%) |
| 2025-11-26 15:57:25,619 - __main__ - INFO - 📈 Epoch progress: 0.23/6 (3.8%) |
| 2025-11-26 15:57:46,533 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (3.9%) |
| 2025-11-26 15:58:07,360 - __main__ - INFO - 📈 Epoch progress: 0.24/6 (4.1%) |
| 2025-11-26 15:58:28,104 - __main__ - INFO - 📈 Epoch progress: 0.25/6 (4.2%) |
| 2025-11-26 15:58:49,607 - __main__ - INFO - 📈 Epoch progress: 0.26/6 (4.3%) |
| 2025-11-26 15:59:10,293 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.4%) |
| 2025-11-26 15:59:30,763 - __main__ - INFO - 📈 Epoch progress: 0.27/6 (4.5%) |
| 2025-11-26 15:59:51,217 - __main__ - INFO - 📈 Epoch progress: 0.28/6 (4.7%) |
| 2025-11-26 16:00:12,027 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.8%) |
| 2025-11-26 16:00:58,821 - __main__ - INFO - 📊 Evaluation 3: BLEU = 0.1186 |
| 2025-11-26 16:00:59,196 - __main__ - INFO - 📊 Evaluation at step 4000 |
| 2025-11-26 16:01:05,960 - __main__ - INFO - |
| 📤 Pushing checkpoint 4000 to HuggingFace Hub... |
| 2025-11-26 16:01:05,961 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 16:01:18,111 - __main__ - INFO - ✅ Checkpoint 4000 pushed successfully! |
| 2025-11-26 16:01:18,115 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-4000 |
| 2025-11-26 16:01:39,069 - __main__ - INFO - 📈 Epoch progress: 0.29/6 (4.9%) |
| 2025-11-26 16:01:59,618 - __main__ - INFO - 📈 Epoch progress: 0.30/6 (5.0%) |
| 2025-11-26 16:02:19,926 - __main__ - INFO - 📈 Epoch progress: 0.31/6 (5.1%) |
| 2025-11-26 16:02:40,522 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.3%) |
| 2025-11-26 16:03:01,310 - __main__ - INFO - 📈 Epoch progress: 0.32/6 (5.4%) |
| 2025-11-26 16:03:22,321 - __main__ - INFO - 📈 Epoch progress: 0.33/6 (5.5%) |
| 2025-11-26 16:03:43,805 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.6%) |
| 2025-11-26 16:04:04,893 - __main__ - INFO - 📈 Epoch progress: 0.34/6 (5.7%) |
| 2025-11-26 16:04:25,380 - __main__ - INFO - 📈 Epoch progress: 0.35/6 (5.9%) |
| 2025-11-26 16:04:46,071 - __main__ - INFO - 📈 Epoch progress: 0.36/6 (6.0%) |
| 2025-11-26 16:05:06,903 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.1%) |
| 2025-11-26 16:05:28,054 - __main__ - INFO - 📈 Epoch progress: 0.37/6 (6.2%) |
| 2025-11-26 16:05:48,374 - __main__ - INFO - 📈 Epoch progress: 0.38/6 (6.3%) |
| 2025-11-26 16:06:09,404 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.5%) |
| 2025-11-26 16:06:30,087 - __main__ - INFO - 📈 Epoch progress: 0.39/6 (6.6%) |
| 2025-11-26 16:06:50,702 - __main__ - INFO - 📈 Epoch progress: 0.40/6 (6.7%) |
| 2025-11-26 16:07:11,672 - __main__ - INFO - 📈 Epoch progress: 0.41/6 (6.8%) |
| 2025-11-26 16:07:32,187 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (6.9%) |
| 2025-11-26 16:07:52,929 - __main__ - INFO - 📈 Epoch progress: 0.42/6 (7.0%) |
| 2025-11-26 16:08:13,223 - __main__ - INFO - 📈 Epoch progress: 0.43/6 (7.2%) |
| 2025-11-26 16:09:01,150 - __main__ - INFO - 📊 Evaluation 4: BLEU = 0.1098 |
| 2025-11-26 16:09:01,557 - __main__ - INFO - 📊 Evaluation at step 6000 |
| 2025-11-26 16:09:06,281 - __main__ - INFO - |
| 📤 Pushing checkpoint 6000 to HuggingFace Hub... |
| 2025-11-26 16:09:06,282 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 16:09:17,586 - __main__ - INFO - ✅ Checkpoint 6000 pushed successfully! |
| 2025-11-26 16:09:17,604 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-6000 |
| 2025-11-26 16:09:37,393 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.3%) |
| 2025-11-26 16:09:58,395 - __main__ - INFO - 📈 Epoch progress: 0.44/6 (7.4%) |
| 2025-11-26 16:10:19,357 - __main__ - INFO - 📈 Epoch progress: 0.45/6 (7.5%) |
| 2025-11-26 16:10:40,495 - __main__ - INFO - 📈 Epoch progress: 0.46/6 (7.6%) |
| 2025-11-26 16:11:01,094 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.8%) |
| 2025-11-26 16:11:22,057 - __main__ - INFO - 📈 Epoch progress: 0.47/6 (7.9%) |
| 2025-11-26 16:11:42,721 - __main__ - INFO - 📈 Epoch progress: 0.48/6 (8.0%) |
| 2025-11-26 16:12:02,880 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.1%) |
| 2025-11-26 16:12:23,276 - __main__ - INFO - 📈 Epoch progress: 0.49/6 (8.2%) |
| 2025-11-26 16:12:43,535 - __main__ - INFO - 📈 Epoch progress: 0.50/6 (8.4%) |
| 2025-11-26 16:13:04,351 - __main__ - INFO - 📈 Epoch progress: 0.51/6 (8.5%) |
| 2025-11-26 16:13:25,533 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.6%) |
| 2025-11-26 16:13:46,002 - __main__ - INFO - 📈 Epoch progress: 0.52/6 (8.7%) |
| 2025-11-26 16:14:06,815 - __main__ - INFO - 📈 Epoch progress: 0.53/6 (8.8%) |
| 2025-11-26 16:14:27,612 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.0%) |
| 2025-11-26 16:14:48,687 - __main__ - INFO - 📈 Epoch progress: 0.54/6 (9.1%) |
| 2025-11-26 16:15:09,813 - __main__ - INFO - 📈 Epoch progress: 0.55/6 (9.2%) |
| 2025-11-26 16:15:30,687 - __main__ - INFO - 📈 Epoch progress: 0.56/6 (9.3%) |
| 2025-11-26 16:15:50,864 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.4%) |
| 2025-11-26 16:16:11,185 - __main__ - INFO - 📈 Epoch progress: 0.57/6 (9.6%) |
| 2025-11-26 16:16:57,438 - __main__ - INFO - 📊 Evaluation 5: BLEU = 0.1175 |
| 2025-11-26 16:16:57,820 - __main__ - INFO - 📊 Evaluation at step 8000 |
| 2025-11-26 16:17:04,619 - __main__ - INFO - |
| 📤 Pushing checkpoint 8000 to HuggingFace Hub... |
| 2025-11-26 16:17:04,620 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 16:17:15,572 - __main__ - INFO - ✅ Checkpoint 8000 pushed successfully! |
| 2025-11-26 16:17:15,574 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-8000 |
| 2025-11-26 16:17:35,586 - __main__ - INFO - 📈 Epoch progress: 0.58/6 (9.7%) |
| 2025-11-26 16:17:56,297 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.8%) |
| 2025-11-26 16:18:17,087 - __main__ - INFO - 📈 Epoch progress: 0.59/6 (9.9%) |
| 2025-11-26 16:18:37,822 - __main__ - INFO - 📈 Epoch progress: 0.60/6 (10.0%) |
| 2025-11-26 16:18:58,361 - __main__ - INFO - 📈 Epoch progress: 0.61/6 (10.2%) |
| 2025-11-26 16:19:18,885 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.3%) |
| 2025-11-26 16:19:39,385 - __main__ - INFO - 📈 Epoch progress: 0.62/6 (10.4%) |
| 2025-11-26 16:19:59,998 - __main__ - INFO - 📈 Epoch progress: 0.63/6 (10.5%) |
| 2025-11-26 16:20:20,640 - __main__ - INFO - 📈 Epoch progress: 0.64/6 (10.6%) |
| 2025-11-26 16:20:41,239 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.8%) |
| 2025-11-26 16:21:01,604 - __main__ - INFO - 📈 Epoch progress: 0.65/6 (10.9%) |
| 2025-11-26 16:21:21,953 - __main__ - INFO - 📈 Epoch progress: 0.66/6 (11.0%) |
| 2025-11-26 16:21:42,831 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.1%) |
| 2025-11-26 16:22:03,702 - __main__ - INFO - 📈 Epoch progress: 0.67/6 (11.2%) |
| 2025-11-26 16:22:23,768 - __main__ - INFO - 📈 Epoch progress: 0.68/6 (11.3%) |
| 2025-11-26 16:22:44,385 - __main__ - INFO - 📈 Epoch progress: 0.69/6 (11.5%) |
| 2025-11-26 16:23:05,353 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.6%) |
| 2025-11-26 16:23:26,172 - __main__ - INFO - 📈 Epoch progress: 0.70/6 (11.7%) |
| 2025-11-26 16:23:46,306 - __main__ - INFO - 📈 Epoch progress: 0.71/6 (11.8%) |
| 2025-11-26 16:24:06,386 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (11.9%) |
| 2025-11-26 16:24:52,276 - __main__ - INFO - 📊 Evaluation 6: BLEU = 0.1201 |
| 2025-11-26 16:24:52,658 - __main__ - INFO - 📊 Evaluation at step 10000 |
| 2025-11-26 16:24:56,162 - __main__ - INFO - |
| 📤 Pushing checkpoint 10000 to HuggingFace Hub... |
| 2025-11-26 16:24:56,163 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 16:25:06,006 - __main__ - INFO - ✅ Checkpoint 10000 pushed successfully! |
| 2025-11-26 16:25:06,015 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-10000 |
| 2025-11-26 16:25:26,160 - __main__ - INFO - 📈 Epoch progress: 0.72/6 (12.1%) |
| 2025-11-26 16:25:47,389 - __main__ - INFO - 📈 Epoch progress: 0.73/6 (12.2%) |
| 2025-11-26 16:26:07,910 - __main__ - INFO - 📈 Epoch progress: 0.74/6 (12.3%) |
| 2025-11-26 16:26:28,628 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.4%) |
| 2025-11-26 16:26:48,827 - __main__ - INFO - 📈 Epoch progress: 0.75/6 (12.5%) |
| 2025-11-26 16:27:09,519 - __main__ - INFO - 📈 Epoch progress: 0.76/6 (12.7%) |
| 2025-11-26 16:27:29,858 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.8%) |
| 2025-11-26 16:27:50,683 - __main__ - INFO - 📈 Epoch progress: 0.77/6 (12.9%) |
| 2025-11-26 16:28:16,023 - __main__ - INFO - 📈 Epoch progress: 0.78/6 (13.0%) |
| 2025-11-26 16:28:37,030 - __main__ - INFO - 📈 Epoch progress: 0.79/6 (13.1%) |
| 2025-11-26 16:28:57,372 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.3%) |
| 2025-11-26 16:29:17,511 - __main__ - INFO - 📈 Epoch progress: 0.80/6 (13.4%) |
| 2025-11-26 16:29:38,325 - __main__ - INFO - 📈 Epoch progress: 0.81/6 (13.5%) |
| 2025-11-26 16:29:59,091 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.6%) |
| 2025-11-26 16:30:19,444 - __main__ - INFO - 📈 Epoch progress: 0.82/6 (13.7%) |
| 2025-11-26 16:30:40,279 - __main__ - INFO - 📈 Epoch progress: 0.83/6 (13.9%) |
| 2025-11-26 16:31:00,631 - __main__ - INFO - 📈 Epoch progress: 0.84/6 (14.0%) |
| 2025-11-26 16:31:21,133 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.1%) |
| 2025-11-26 16:31:41,554 - __main__ - INFO - 📈 Epoch progress: 0.85/6 (14.2%) |
| 2025-11-26 16:32:01,981 - __main__ - INFO - 📈 Epoch progress: 0.86/6 (14.3%) |
| 2025-11-26 16:32:55,177 - __main__ - INFO - 📊 Evaluation 7: BLEU = 0.1172 |
| 2025-11-26 16:32:56,101 - __main__ - INFO - 📊 Evaluation at step 12000 |
| 2025-11-26 16:33:00,167 - __main__ - INFO - |
| 📤 Pushing checkpoint 12000 to HuggingFace Hub... |
| 2025-11-26 16:33:00,167 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 16:33:10,375 - __main__ - INFO - ✅ Checkpoint 12000 pushed successfully! |
| 2025-11-26 16:33:10,376 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-12000 |
| 2025-11-26 16:33:31,660 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.5%) |
| 2025-11-26 16:33:51,937 - __main__ - INFO - 📈 Epoch progress: 0.87/6 (14.6%) |
| 2025-11-26 16:34:12,412 - __main__ - INFO - 📈 Epoch progress: 0.88/6 (14.7%) |
| 2025-11-26 16:34:33,310 - __main__ - INFO - 📈 Epoch progress: 0.89/6 (14.8%) |
| 2025-11-26 16:34:53,571 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (14.9%) |
| 2025-11-26 16:35:14,347 - __main__ - INFO - 📈 Epoch progress: 0.90/6 (15.1%) |
| 2025-11-26 16:35:34,720 - __main__ - INFO - 📈 Epoch progress: 0.91/6 (15.2%) |
| 2025-11-26 16:35:55,386 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.3%) |
| 2025-11-26 16:36:15,968 - __main__ - INFO - 📈 Epoch progress: 0.92/6 (15.4%) |
| 2025-11-26 16:36:36,705 - __main__ - INFO - 📈 Epoch progress: 0.93/6 (15.5%) |
| 2025-11-26 16:36:57,025 - __main__ - INFO - 📈 Epoch progress: 0.94/6 (15.6%) |
| 2025-11-26 16:37:17,355 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.8%) |
| 2025-11-26 16:37:37,781 - __main__ - INFO - 📈 Epoch progress: 0.95/6 (15.9%) |
| 2025-11-26 16:37:58,781 - __main__ - INFO - 📈 Epoch progress: 0.96/6 (16.0%) |
| 2025-11-26 16:38:19,375 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.1%) |
| 2025-11-26 16:38:40,368 - __main__ - INFO - 📈 Epoch progress: 0.97/6 (16.2%) |
| 2025-11-26 16:39:00,906 - __main__ - INFO - 📈 Epoch progress: 0.98/6 (16.4%) |
| 2025-11-26 16:39:22,058 - __main__ - INFO - 📈 Epoch progress: 0.99/6 (16.5%) |
| 2025-11-26 16:39:42,516 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.6%) |
| 2025-11-26 16:40:04,615 - __main__ - INFO - 📈 Epoch progress: 1.00/6 (16.7%) |
| 2025-11-26 16:40:52,808 - __main__ - INFO - 📊 Evaluation 8: BLEU = 0.1178 |
| 2025-11-26 16:40:53,273 - __main__ - INFO - 📊 Evaluation at step 14000 |
| 2025-11-26 16:40:57,556 - __main__ - INFO - |
| 📤 Pushing checkpoint 14000 to HuggingFace Hub... |
| 2025-11-26 16:40:57,557 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 16:41:12,071 - __main__ - INFO - ✅ Checkpoint 14000 pushed successfully! |
| 2025-11-26 16:41:12,077 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-14000 |
| 2025-11-26 16:41:32,297 - __main__ - INFO - 📈 Epoch progress: 1.01/6 (16.8%) |
| 2025-11-26 16:41:53,076 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.0%) |
| 2025-11-26 16:42:13,244 - __main__ - INFO - 📈 Epoch progress: 1.02/6 (17.1%) |
| 2025-11-26 16:42:33,859 - __main__ - INFO - 📈 Epoch progress: 1.03/6 (17.2%) |
| 2025-11-26 16:42:54,826 - __main__ - INFO - 📈 Epoch progress: 1.04/6 (17.3%) |
| 2025-11-26 16:43:15,509 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.4%) |
| 2025-11-26 16:43:36,088 - __main__ - INFO - 📈 Epoch progress: 1.05/6 (17.6%) |
| 2025-11-26 16:43:57,218 - __main__ - INFO - 📈 Epoch progress: 1.06/6 (17.7%) |
| 2025-11-26 16:44:17,864 - __main__ - INFO - 📈 Epoch progress: 1.07/6 (17.8%) |
| 2025-11-26 16:44:38,282 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (17.9%) |
| 2025-11-26 16:44:58,756 - __main__ - INFO - 📈 Epoch progress: 1.08/6 (18.0%) |
| 2025-11-26 16:45:19,109 - __main__ - INFO - 📈 Epoch progress: 1.09/6 (18.2%) |
| 2025-11-26 16:45:39,663 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.3%) |
| 2025-11-26 16:46:00,369 - __main__ - INFO - 📈 Epoch progress: 1.10/6 (18.4%) |
| 2025-11-26 16:46:21,166 - __main__ - INFO - 📈 Epoch progress: 1.11/6 (18.5%) |
| 2025-11-26 16:46:41,702 - __main__ - INFO - 📈 Epoch progress: 1.12/6 (18.6%) |
| 2025-11-26 16:47:02,646 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.8%) |
| 2025-11-26 16:47:23,510 - __main__ - INFO - 📈 Epoch progress: 1.13/6 (18.9%) |
| 2025-11-26 16:47:44,422 - __main__ - INFO - 📈 Epoch progress: 1.14/6 (19.0%) |
| 2025-11-26 16:48:04,903 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.1%) |
| 2025-11-26 16:48:53,083 - __main__ - INFO - 📊 Evaluation 9: BLEU = 0.1183 |
| 2025-11-26 16:48:53,626 - __main__ - INFO - 📊 Evaluation at step 16000 |
| 2025-11-26 16:48:57,078 - __main__ - INFO - |
| 📤 Pushing checkpoint 16000 to HuggingFace Hub... |
| 2025-11-26 16:48:57,078 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 16:49:06,782 - __main__ - INFO - ✅ Checkpoint 16000 pushed successfully! |
| 2025-11-26 16:49:06,784 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-16000 |
| 2025-11-26 16:49:26,693 - __main__ - INFO - 📈 Epoch progress: 1.15/6 (19.2%) |
| 2025-11-26 16:49:47,398 - __main__ - INFO - 📈 Epoch progress: 1.16/6 (19.4%) |
| 2025-11-26 16:50:07,998 - __main__ - INFO - 📈 Epoch progress: 1.17/6 (19.5%) |
| 2025-11-26 16:50:28,528 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.6%) |
| 2025-11-26 16:50:49,162 - __main__ - INFO - 📈 Epoch progress: 1.18/6 (19.7%) |
| 2025-11-26 16:51:10,379 - __main__ - INFO - 📈 Epoch progress: 1.19/6 (19.8%) |
| 2025-11-26 16:51:31,188 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (19.9%) |
| 2025-11-26 16:51:51,598 - __main__ - INFO - 📈 Epoch progress: 1.20/6 (20.1%) |
| 2025-11-26 16:52:12,375 - __main__ - INFO - 📈 Epoch progress: 1.21/6 (20.2%) |
| 2025-11-26 16:52:32,087 - __main__ - INFO - 📈 Epoch progress: 1.22/6 (20.3%) |
| 2025-11-26 16:52:52,896 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.4%) |
| 2025-11-26 16:53:13,262 - __main__ - INFO - 📈 Epoch progress: 1.23/6 (20.5%) |
| 2025-11-26 16:53:33,806 - __main__ - INFO - 📈 Epoch progress: 1.24/6 (20.7%) |
| 2025-11-26 16:53:54,820 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.8%) |
| 2025-11-26 16:54:15,617 - __main__ - INFO - 📈 Epoch progress: 1.25/6 (20.9%) |
| 2025-11-26 16:54:36,064 - __main__ - INFO - 📈 Epoch progress: 1.26/6 (21.0%) |
| 2025-11-26 16:54:57,165 - __main__ - INFO - 📈 Epoch progress: 1.27/6 (21.1%) |
| 2025-11-26 16:55:17,640 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.3%) |
| 2025-11-26 16:55:38,662 - __main__ - INFO - 📈 Epoch progress: 1.28/6 (21.4%) |
| 2025-11-26 16:55:59,432 - __main__ - INFO - 📈 Epoch progress: 1.29/6 (21.5%) |
| 2025-11-26 16:56:44,724 - __main__ - INFO - 📊 Evaluation 10: BLEU = 0.1219 |
| 2025-11-26 16:56:45,222 - __main__ - INFO - 📊 Evaluation at step 18000 |
| 2025-11-26 16:56:49,564 - __main__ - INFO - |
| 📤 Pushing checkpoint 18000 to HuggingFace Hub... |
| 2025-11-26 16:56:49,565 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 16:57:00,156 - __main__ - INFO - ✅ Checkpoint 18000 pushed successfully! |
| 2025-11-26 16:57:00,157 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-18000 |
| 2025-11-26 16:57:20,382 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.6%) |
| 2025-11-26 16:57:41,369 - __main__ - INFO - 📈 Epoch progress: 1.30/6 (21.7%) |
| 2025-11-26 16:58:01,591 - __main__ - INFO - 📈 Epoch progress: 1.31/6 (21.9%) |
| 2025-11-26 16:58:22,170 - __main__ - INFO - 📈 Epoch progress: 1.32/6 (22.0%) |
| 2025-11-26 16:58:43,488 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.1%) |
| 2025-11-26 16:59:04,061 - __main__ - INFO - 📈 Epoch progress: 1.33/6 (22.2%) |
| 2025-11-26 16:59:24,749 - __main__ - INFO - 📈 Epoch progress: 1.34/6 (22.3%) |
| 2025-11-26 16:59:45,462 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.5%) |
| 2025-11-26 17:00:06,941 - __main__ - INFO - 📈 Epoch progress: 1.35/6 (22.6%) |
| 2025-11-26 17:00:27,513 - __main__ - INFO - 📈 Epoch progress: 1.36/6 (22.7%) |
| 2025-11-26 17:00:47,989 - __main__ - INFO - 📈 Epoch progress: 1.37/6 (22.8%) |
| 2025-11-26 17:01:08,467 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (22.9%) |
| 2025-11-26 17:01:29,023 - __main__ - INFO - 📈 Epoch progress: 1.38/6 (23.1%) |
| 2025-11-26 17:01:49,659 - __main__ - INFO - 📈 Epoch progress: 1.39/6 (23.2%) |
| 2025-11-26 17:02:10,319 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.3%) |
| 2025-11-26 17:02:31,722 - __main__ - INFO - 📈 Epoch progress: 1.40/6 (23.4%) |
| 2025-11-26 17:02:54,391 - __main__ - INFO - 📈 Epoch progress: 1.41/6 (23.5%) |
| 2025-11-26 17:03:15,204 - __main__ - INFO - 📈 Epoch progress: 1.42/6 (23.7%) |
| 2025-11-26 17:03:35,749 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.8%) |
| 2025-11-26 17:03:56,077 - __main__ - INFO - 📈 Epoch progress: 1.43/6 (23.9%) |
| 2025-11-26 17:04:43,169 - __main__ - INFO - 📊 Evaluation 11: BLEU = 0.1225 |
| 2025-11-26 17:04:43,613 - __main__ - INFO - 📊 Evaluation at step 20000 |
| 2025-11-26 17:04:52,366 - __main__ - INFO - |
| 📤 Pushing checkpoint 20000 to HuggingFace Hub... |
| 2025-11-26 17:04:52,367 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 17:05:03,979 - __main__ - INFO - ✅ Checkpoint 20000 pushed successfully! |
| 2025-11-26 17:05:03,980 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-20000 |
| 2025-11-26 17:05:24,366 - __main__ - INFO - 📈 Epoch progress: 1.44/6 (24.0%) |
| 2025-11-26 17:05:44,722 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.1%) |
| 2025-11-26 17:06:05,254 - __main__ - INFO - 📈 Epoch progress: 1.45/6 (24.2%) |
| 2025-11-26 17:06:26,544 - __main__ - INFO - 📈 Epoch progress: 1.46/6 (24.4%) |
| 2025-11-26 17:06:47,089 - __main__ - INFO - 📈 Epoch progress: 1.47/6 (24.5%) |
| 2025-11-26 17:07:07,849 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.6%) |
| 2025-11-26 17:07:28,326 - __main__ - INFO - 📈 Epoch progress: 1.48/6 (24.7%) |
| 2025-11-26 17:07:49,542 - __main__ - INFO - 📈 Epoch progress: 1.49/6 (24.8%) |
| 2025-11-26 17:08:09,977 - __main__ - INFO - 📈 Epoch progress: 1.50/6 (25.0%) |
| 2025-11-26 17:08:30,453 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.1%) |
| 2025-11-26 17:08:51,842 - __main__ - INFO - 📈 Epoch progress: 1.51/6 (25.2%) |
| 2025-11-26 17:09:12,306 - __main__ - INFO - 📈 Epoch progress: 1.52/6 (25.3%) |
| 2025-11-26 17:09:32,993 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.4%) |
| 2025-11-26 17:09:53,319 - __main__ - INFO - 📈 Epoch progress: 1.53/6 (25.6%) |
| 2025-11-26 17:10:13,578 - __main__ - INFO - 📈 Epoch progress: 1.54/6 (25.7%) |
| 2025-11-26 17:10:34,380 - __main__ - INFO - 📈 Epoch progress: 1.55/6 (25.8%) |
| 2025-11-26 17:10:54,895 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (25.9%) |
| 2025-11-26 17:11:15,327 - __main__ - INFO - 📈 Epoch progress: 1.56/6 (26.0%) |
| 2025-11-26 17:11:35,577 - __main__ - INFO - 📈 Epoch progress: 1.57/6 (26.2%) |
| 2025-11-26 17:11:57,376 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.3%) |
| 2025-11-26 17:12:52,830 - __main__ - INFO - 📊 Evaluation 12: BLEU = 0.1133 |
| 2025-11-26 17:12:53,225 - __main__ - INFO - 📊 Evaluation at step 22000 |
| 2025-11-26 17:12:57,360 - __main__ - INFO - |
| 📤 Pushing checkpoint 22000 to HuggingFace Hub... |
| 2025-11-26 17:12:57,361 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 17:13:06,308 - __main__ - INFO - ✅ Checkpoint 22000 pushed successfully! |
| 2025-11-26 17:13:06,314 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-22000 |
| 2025-11-26 17:13:26,399 - __main__ - INFO - 📈 Epoch progress: 1.58/6 (26.4%) |
| 2025-11-26 17:13:47,465 - __main__ - INFO - 📈 Epoch progress: 1.59/6 (26.5%) |
| 2025-11-26 17:14:08,586 - __main__ - INFO - 📈 Epoch progress: 1.60/6 (26.6%) |
| 2025-11-26 17:14:28,839 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.8%) |
| 2025-11-26 17:14:50,346 - __main__ - INFO - 📈 Epoch progress: 1.61/6 (26.9%) |
| 2025-11-26 17:15:10,980 - __main__ - INFO - 📈 Epoch progress: 1.62/6 (27.0%) |
| 2025-11-26 17:15:31,737 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.1%) |
| 2025-11-26 17:15:52,488 - __main__ - INFO - 📈 Epoch progress: 1.63/6 (27.2%) |
| 2025-11-26 17:16:13,006 - __main__ - INFO - 📈 Epoch progress: 1.64/6 (27.4%) |
| 2025-11-26 17:16:34,452 - __main__ - INFO - 📈 Epoch progress: 1.65/6 (27.5%) |
| 2025-11-26 17:16:55,065 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.6%) |
| 2025-11-26 17:17:15,489 - __main__ - INFO - 📈 Epoch progress: 1.66/6 (27.7%) |
| 2025-11-26 17:17:35,891 - __main__ - INFO - 📈 Epoch progress: 1.67/6 (27.8%) |
| 2025-11-26 17:17:56,335 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.0%) |
| 2025-11-26 17:18:17,309 - __main__ - INFO - 📈 Epoch progress: 1.68/6 (28.1%) |
| 2025-11-26 17:18:37,633 - __main__ - INFO - 📈 Epoch progress: 1.69/6 (28.2%) |
| 2025-11-26 17:18:58,357 - __main__ - INFO - 📈 Epoch progress: 1.70/6 (28.3%) |
| 2025-11-26 17:19:20,368 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.4%) |
| 2025-11-26 17:19:41,773 - __main__ - INFO - 📈 Epoch progress: 1.71/6 (28.5%) |
| 2025-11-26 17:20:02,469 - __main__ - INFO - 📈 Epoch progress: 1.72/6 (28.7%) |
| 2025-11-26 17:20:45,791 - __main__ - INFO - 📊 Evaluation 13: BLEU = 0.1197 |
| 2025-11-26 17:20:46,181 - __main__ - INFO - 📊 Evaluation at step 24000 |
| 2025-11-26 17:20:49,966 - __main__ - INFO - |
| 📤 Pushing checkpoint 24000 to HuggingFace Hub... |
| 2025-11-26 17:20:49,966 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 17:21:00,484 - __main__ - INFO - ✅ Checkpoint 24000 pushed successfully! |
| 2025-11-26 17:21:00,486 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-24000 |
| 2025-11-26 17:21:21,002 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.8%) |
| 2025-11-26 17:21:41,788 - __main__ - INFO - 📈 Epoch progress: 1.73/6 (28.9%) |
| 2025-11-26 17:22:02,130 - __main__ - INFO - 📈 Epoch progress: 1.74/6 (29.0%) |
| 2025-11-26 17:22:23,172 - __main__ - INFO - 📈 Epoch progress: 1.75/6 (29.1%) |
| 2025-11-26 17:22:43,451 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.3%) |
| 2025-11-26 17:23:04,553 - __main__ - INFO - 📈 Epoch progress: 1.76/6 (29.4%) |
| 2025-11-26 17:23:25,507 - __main__ - INFO - 📈 Epoch progress: 1.77/6 (29.5%) |
| 2025-11-26 17:23:46,026 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.6%) |
| 2025-11-26 17:24:07,282 - __main__ - INFO - 📈 Epoch progress: 1.78/6 (29.7%) |
| 2025-11-26 17:24:28,108 - __main__ - INFO - 📈 Epoch progress: 1.79/6 (29.9%) |
| 2025-11-26 17:24:48,818 - __main__ - INFO - 📈 Epoch progress: 1.80/6 (30.0%) |
| 2025-11-26 17:25:09,138 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.1%) |
| 2025-11-26 17:25:29,625 - __main__ - INFO - 📈 Epoch progress: 1.81/6 (30.2%) |
| 2025-11-26 17:25:50,234 - __main__ - INFO - 📈 Epoch progress: 1.82/6 (30.3%) |
| 2025-11-26 17:26:10,569 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.5%) |
| 2025-11-26 17:26:31,382 - __main__ - INFO - 📈 Epoch progress: 1.83/6 (30.6%) |
| 2025-11-26 17:26:51,848 - __main__ - INFO - 📈 Epoch progress: 1.84/6 (30.7%) |
| 2025-11-26 17:27:12,730 - __main__ - INFO - 📈 Epoch progress: 1.85/6 (30.8%) |
| 2025-11-26 17:27:33,559 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (30.9%) |
| 2025-11-26 17:27:54,145 - __main__ - INFO - 📈 Epoch progress: 1.86/6 (31.1%) |
| 2025-11-26 17:28:37,769 - __main__ - INFO - 📊 Evaluation 14: BLEU = 0.1220 |
| 2025-11-26 17:28:38,281 - __main__ - INFO - 📊 Evaluation at step 26000 |
| 2025-11-26 17:28:42,091 - __main__ - INFO - |
| 📤 Pushing checkpoint 26000 to HuggingFace Hub... |
| 2025-11-26 17:28:42,092 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 17:28:51,807 - __main__ - INFO - ✅ Checkpoint 26000 pushed successfully! |
| 2025-11-26 17:28:51,815 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-26000 |
| 2025-11-26 17:29:12,107 - __main__ - INFO - 📈 Epoch progress: 1.87/6 (31.2%) |
| 2025-11-26 17:29:33,590 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.3%) |
| 2025-11-26 17:29:54,623 - __main__ - INFO - 📈 Epoch progress: 1.88/6 (31.4%) |
| 2025-11-26 17:30:15,939 - __main__ - INFO - 📈 Epoch progress: 1.89/6 (31.5%) |
| 2025-11-26 17:30:37,067 - __main__ - INFO - 📈 Epoch progress: 1.90/6 (31.7%) |
| 2025-11-26 17:30:58,061 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.8%) |
| 2025-11-26 17:31:18,541 - __main__ - INFO - 📈 Epoch progress: 1.91/6 (31.9%) |
| 2025-11-26 17:31:39,182 - __main__ - INFO - 📈 Epoch progress: 1.92/6 (32.0%) |
| 2025-11-26 17:31:59,476 - __main__ - INFO - 📈 Epoch progress: 1.93/6 (32.1%) |
| 2025-11-26 17:32:20,115 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.3%) |
| 2025-11-26 17:32:40,678 - __main__ - INFO - 📈 Epoch progress: 1.94/6 (32.4%) |
| 2025-11-26 17:33:01,358 - __main__ - INFO - 📈 Epoch progress: 1.95/6 (32.5%) |
| 2025-11-26 17:33:22,398 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.6%) |
| 2025-11-26 17:33:43,021 - __main__ - INFO - 📈 Epoch progress: 1.96/6 (32.7%) |
| 2025-11-26 17:34:03,906 - __main__ - INFO - 📈 Epoch progress: 1.97/6 (32.8%) |
| 2025-11-26 17:34:25,041 - __main__ - INFO - 📈 Epoch progress: 1.98/6 (33.0%) |
| 2025-11-26 17:34:45,997 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.1%) |
| 2025-11-26 17:35:07,381 - __main__ - INFO - 📈 Epoch progress: 1.99/6 (33.2%) |
| 2025-11-26 17:35:28,654 - __main__ - INFO - 📈 Epoch progress: 2.00/6 (33.3%) |
| 2025-11-26 17:35:50,983 - __main__ - INFO - 📈 Epoch progress: 2.01/6 (33.4%) |
| 2025-11-26 17:36:28,715 - __main__ - INFO - 📊 Evaluation 15: BLEU = 0.1212 |
| 2025-11-26 17:36:29,111 - __main__ - INFO - 📊 Evaluation at step 28000 |
| 2025-11-26 17:36:36,019 - __main__ - INFO - |
| 📤 Pushing checkpoint 28000 to HuggingFace Hub... |
| 2025-11-26 17:36:36,020 - __main__ - INFO - Repository: brendaogutu/mul-sw-en-translation-phase2 |
| 2025-11-26 17:36:49,430 - __main__ - INFO - ✅ Checkpoint 28000 pushed successfully! |
| 2025-11-26 17:36:49,434 - __main__ - INFO - 🔗 View at: https://huggingface.co/brendaogutu/mul-sw-en-translation-phase2/tree/main/checkpoints/checkpoint-28000 |
| 2025-11-26 17:36:56,016 - __main__ - INFO - ✅ MLflow: Training finished - status_code=2.0 (FINISHED) |
| 2025-11-26 17:36:56,297 - __main__ - INFO - |
| 💾 Saving model to models/finetuned-sw-en-phase2 |
| 2025-11-26 17:36:57,686 - __main__ - INFO - ✅ Training complete! |
| 2025-11-26 17:36:57,687 - __main__ - INFO - |
| 📊 STEP 3: Final Evaluation |
| 2025-11-26 17:36:58,087 - __main__ - INFO - ✅ MLflow: Final evaluation starting - status_code=3.0 |
| 2025-11-26 17:36:58,089 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 17:36:58,090 - __main__ - INFO - 📊 Running FINAL evaluation on test set... |
| 2025-11-26 17:36:58,091 - __main__ - INFO - ====================================================================== |
| 2025-11-26 17:36:58,091 - __main__ - INFO - Tokenizing test dataset... |
| 2025-11-26 17:37:00,087 - datasets.fingerprint - WARNING - Parameter 'function'=<bound method TranslationTrainer.preprocess_function of <__main__.TranslationTrainer object at 0x73e428e41a00>> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown. |
| 2025-11-26 18:55:14,071 - __main__ - INFO - 📊 Evaluation 16: BLEU = 0.1289 |
| 2025-11-26 18:55:14,655 - __main__ - INFO - 📊 Evaluation at step 28000 |
| 2025-11-26 18:55:14,800 - __main__ - INFO - 📊 Computing COMET on test set... |
| 2025-11-26 18:55:14,860 - __main__ - INFO - Computing predictions for COMET (1000 samples, batch_size=32)... |
| 2025-11-26 18:55:14,861 - __main__ - INFO - Batch 1/32... |
| 2025-11-26 18:55:18,346 - __main__ - INFO - Batch 5/32... |
| 2025-11-26 18:55:19,409 - __main__ - INFO - Batch 10/32... |
| 2025-11-26 18:55:20,560 - __main__ - INFO - Batch 15/32... |
| 2025-11-26 18:55:21,821 - __main__ - INFO - Batch 20/32... |
| 2025-11-26 18:55:22,971 - __main__ - INFO - Batch 25/32... |
| 2025-11-26 18:55:24,126 - __main__ - INFO - Batch 30/32... |
| 2025-11-26 18:55:24,716 - __main__ - INFO - Running COMET-QE model... |
| 2025-11-26 18:55:24,731 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-26 18:55:24,739 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-26 18:55:24,740 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-26 18:55:24,741 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-26 18:55:29,735 - __main__ - INFO - ✅ COMET-QE: 0.1763 ± 0.2628 |
| 2025-11-26 18:55:29,736 - __main__ - INFO - |
| ✅ FINAL Test Results: |
| 2025-11-26 18:55:29,737 - __main__ - INFO - BLEU: 0.1289 |
| 2025-11-26 18:55:29,738 - __main__ - INFO - chrF: 32.25 |
| 2025-11-26 18:55:29,738 - __main__ - INFO - COMET-QE: 0.1763 |
| 2025-11-26 18:55:29,738 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 18:55:29,739 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 18:55:29,740 - __main__ - INFO - 📊 Evaluating on DOMAIN data (call transcriptions)... |
| 2025-11-26 18:55:29,741 - __main__ - INFO - ====================================================================== |
| 2025-11-26 18:55:29,742 - __main__ - INFO - Evaluating on 5 domain samples (batch_size=32)... |
| 2025-11-26 18:55:29,743 - __main__ - INFO - Translating batch 1/1 (samples 1-5)... |
| 2025-11-26 18:55:30,630 - __main__ - INFO - ✅ Domain BLEU: 0.0000 |
| 2025-11-26 18:55:30,741 - __main__ - INFO - ✅ Domain chrF: 1.18 |
| 2025-11-26 18:55:30,742 - __main__ - INFO - Computing COMET-QE... |
| 2025-11-26 18:55:30,748 - pytorch_lightning.utilities.rank_zero - INFO - 💡 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry. |
| 2025-11-26 18:55:30,757 - pytorch_lightning.utilities.rank_zero - INFO - GPU available: True (cuda), used: True |
| 2025-11-26 18:55:30,757 - pytorch_lightning.utilities.rank_zero - INFO - TPU available: False, using: 0 TPU cores |
| 2025-11-26 18:55:30,759 - pytorch_lightning.accelerators.cuda - INFO - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] |
| 2025-11-26 18:55:33,071 - __main__ - INFO - ✅ Domain COMET-QE: 0.0000 |
| 2025-11-26 18:55:33,072 - __main__ - INFO - |
| ✅ Domain Evaluation Results: |
| 2025-11-26 18:55:33,072 - __main__ - INFO - BLEU: 0.0000 |
| 2025-11-26 18:55:33,073 - __main__ - INFO - chrF: 1.18 |
| 2025-11-26 18:55:33,073 - __main__ - INFO - COMET-QE: 0.0000 |
| 2025-11-26 18:55:33,074 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 18:55:33,091 - __main__ - INFO - 💾 Final results saved to models/finetuned-sw-en-phase2/test_results.json |
| 2025-11-26 18:55:33,115 - __main__ - INFO - ✅ Verified test_results.json with 14 metrics |
| 2025-11-26 18:55:33,117 - __main__ - INFO - |
| 📊 Logging all metrics to TensorBoard... |
| 2025-11-26 18:55:33,454 - __main__ - INFO - ✅ All metrics logged to TensorBoard |
| 2025-11-26 18:55:33,455 - __main__ - INFO - Baseline metrics: 13 |
| 2025-11-26 18:55:33,456 - __main__ - INFO - Final test metrics: 10 |
| 2025-11-26 18:55:33,457 - __main__ - INFO - Final domain metrics: 4 |
| 2025-11-26 18:55:33,457 - __main__ - INFO - |
| ====================================================================== |
| 2025-11-26 18:55:33,458 - __main__ - INFO - 📊 BASELINE vs FINE-TUNED Comparison |
| 2025-11-26 18:55:33,458 - __main__ - INFO - ====================================================================== |
| 2025-11-26 18:55:33,459 - __main__ - INFO - |
| 📈 Test Set Metrics: |
| 2025-11-26 18:55:33,459 - __main__ - INFO - Metric Baseline Fine-tuned Improvement |
| 2025-11-26 18:55:33,460 - __main__ - INFO - ------------------------------------------------------------ |
| 2025-11-26 18:55:33,460 - __main__ - INFO - eval_bleu 0.0000 0.1289 N/A |
| 2025-11-26 18:55:33,461 - __main__ - INFO - eval_chrf 0.0000 32.2491 N/A |
| 2025-11-26 18:55:33,462 - __main__ - INFO - eval_comet 0.0000 0.1763 N/A |
| 2025-11-26 18:55:33,463 - __main__ - INFO - |
| 📈 Domain Metrics (Real Call Transcriptions): |
| 2025-11-26 18:55:33,463 - __main__ - INFO - Metric Baseline Fine-tuned Improvement |
| 2025-11-26 18:55:33,463 - __main__ - INFO - ------------------------------------------------------------ |
| 2025-11-26 18:55:33,464 - __main__ - INFO - domain_bleu 0.0000 0.0000 +1417.7% |
| 2025-11-26 18:55:33,465 - __main__ - INFO - domain_chrf 1.2250 1.1786 -3.8% |
| 2025-11-26 18:55:33,466 - __main__ - INFO - domain_comet 0.0001 0.0000 -26.2% |
| 2025-11-26 18:55:33,466 - __main__ - INFO - ====================================================================== |
| |
| 2025-11-26 18:55:33,938 - __main__ - INFO - ✅ Logged 14 final metrics to MLflow |
| 2025-11-26 18:55:33,939 - __main__ - INFO - Visit MLflow: https://proceedings-graphic-pressed-responded.trycloudflare.com |
| 2025-11-26 18:55:33,940 - __main__ - INFO - 📊 Captured 576 training log entries |
| 2025-11-26 18:55:33,967 - __main__ - INFO - 💾 Saved comprehensive metrics to models/finetuned-sw-en-phase2/training_metrics.json |
| 2025-11-26 18:55:33,983 - __main__ - INFO - 💾 Saved training config to models/finetuned-sw-en-phase2/training_config_used.json |
| |
