[INFO|trainer.py:2956] 2022-12-12 12:04:03,940 >> ***** Running Evaluation *****

[INFO|trainer.py:2958] 2022-12-12 12:04:03,940 >> Num examples = 1696
[INFO|trainer.py:2961] 2022-12-12 12:04:03,941 >> Batch size = 16
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 106/106 [18:20<00:00, 10.38s/it]
***** eval metrics *****
epoch = 12.44
eval_loss = 0.3801
eval_runtime = 0:18:30.86
eval_samples_per_second = 1.527
eval_steps_per_second = 0.095
eval_wer = 11.1999

***** eval metrics *****
epoch = 12.44
eval_loss = 0.3801
eval_runtime = 0:18:30.86
eval_samples_per_second = 1.527
eval_steps_per_second = 0.095
eval_wer = 11.1999

Mon Dec 12 12:32:15 UTC 2022

.ipynb_checkpoints/README-checkpoint.md

@@ -1,85 +0,0 @@
----
-language:
-- el
-license: apache-2.0
-tags:
-- whisper-event
-- generated_from_trainer
-- hf-asr-leaderboard
-datasets:
-- mozilla-foundation/common_voice_11_0
-- google/fleurs
-metrics:
-- wer
-model-index:
-- name: Whisper Medium El Greco
-  results:
-  - task:
-      name: Automatic Speech Recognition
-      type: automatic-speech-recognition
-    dataset:
-      name: Common Voice 11.0
-      type: mozilla-foundation/common_voice_11_0
-      config: el
-      split: test
-      args: el
-    metrics:
-    - name: Wer
-      type: wer
-      value: 13.976597325408619
----
-
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# Whisper Medium El - Greek One
-
-This model is a fine-tuned version of [openai/whisper-medium](https://huggingface.co/openai/whisper-medium) on the Common Voice 11.0 dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.4707
-- Wer: 13.9766
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 20
-- eval_batch_size: 8
-- seed: 42
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 500
-- training_steps: 5000
-- mixed_precision_training: Native AMP
-
-### Training results
-
-| Training Loss | Epoch | Step | Validation Loss | Wer     |
-|:-------------:|:-----:|:----:|:---------------:|:-------:|
-| 0.0036        | 10.01 | 1000 | 0.4461          | 15.9082 |
-| 0.0001        | 20.02 | 2000 | 0.4250          | 14.5245 |
-| 0.0           | 31.0  | 3000 | 0.4526          | 14.1902 |
-| 0.0           | 41.01 | 4000 | 0.4657          | 14.1252 |
-| 0.0           | 52.0  | 5000 | 0.4707          | 13.9766 |
-
-
-### Framework versions
-
-- Transformers 4.26.0.dev0
-- Pytorch 1.13.0+cu117
-- Datasets 2.7.1.dev0
-- Tokenizers 0.13.2

README.md

@@ -3,30 +3,38 @@ language:
 - el
 license: apache-2.0
 tags:
-- hf-asr-leaderboard, whisper-medium, mozilla-foundation/common_voice_11_0, greek,
-  whisper-event
+- whisper-event
 - generated_from_trainer
 datasets:
-- mozilla-foundation/common_voice_11_0
+- mozilla-foundation/common_voice_11_0,google/fleurs
+metrics:
+- wer
 model-index:
-- name: Whisper Medium El Greco Greek
-  results: []
+- name: Whisper Medium El Greco
+  results:
+  - task:
+      name: Automatic Speech Recognition
+      type: automatic-speech-recognition
+    dataset:
+      name: mozilla-foundation/common_voice_11_0,google/fleurs el,el_gr
+      type: mozilla-foundation/common_voice_11_0,google/fleurs
+      config: null
+      split: None
+    metrics:
+    - name: Wer
+      type: wer
+      value: 11.199851411589897
 ---
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
-# Whisper Medium El Greco Greek
+# Whisper Medium El Greco
 
-This model is a fine-tuned version of [openai/whisper-medium](https://huggingface.co/openai/whisper-medium) on the Common Voice 11.0 dataset.
+This model is a fine-tuned version of [emilios/whisper-medium-el](https://huggingface.co/emilios/whisper-medium-el) on the mozilla-foundation/common_voice_11_0,google/fleurs el,el_gr dataset.
 It achieves the following results on the evaluation set:
-- eval_loss: 0.3924
-- eval_wer: 12.4443
-- eval_runtime: 1211.1631
-- eval_samples_per_second: 1.4
-- eval_steps_per_second: 0.088
-- epoch: 4.04
-- step: 5000
+- Loss: 0.3801
+- Wer: 11.1999
 
 ## Model description
 
@@ -55,6 +63,17 @@ The following hyperparameters were used during training:
 - training_steps: 5000
 - mixed_precision_training: Native AMP
 
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Wer     |
+|:-------------:|:-----:|:----:|:---------------:|:-------:|
+| 0.0176        | 2.49  | 1000 | 0.2945          | 12.6114 |
+| 0.0064        | 4.98  | 2000 | 0.3423          | 12.2307 |
+| 0.0022        | 7.46  | 3000 | 0.3632          | 11.5899 |
+| 0.0014        | 9.95  | 4000 | 0.3788          | 11.2556 |
+| 0.0008        | 12.44 | 5000 | 0.3801          | 11.1999 |
+
+
 ### Framework versions
 
 - Transformers 4.26.0.dev0

all_results.json

@@ -0,0 +1,12 @@
+{
+    "epoch": 12.44,
+    "eval_loss": 0.38012266159057617,
+    "eval_runtime": 1110.8612,
+    "eval_samples_per_second": 1.527,
+    "eval_steps_per_second": 0.095,
+    "eval_wer": 11.199851411589897,
+    "train_loss": 0.020854657278954983,
+    "train_runtime": 27142.1177,
+    "train_samples_per_second": 5.895,
+    "train_steps_per_second": 0.184
+}

config.json

@@ -34,7 +34,6 @@
   "num_mel_bins": 80,
   "pad_token_id": 50257,
   "scale_embedding": false,
-  "suppress_tokens": [],
   "torch_dtype": "float32",
   "transformers_version": "4.26.0.dev0",
   "use_cache": false,

e3_interleaving-cl.ipynb

@@ -120,106 +120,68 @@
      "output_type": "stream",
      "text": [
       "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Requirement already satisfied: pip in ./.local/lib/python3.8/site-packages (22.3.1)\n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Requirement already satisfied: numpy<1.23.0 in ./.local/lib/python3.8/site-packages (1.22.4)\n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Collecting torch\n",
-      "  Using cached torch-1.13.0-cp38-cp38-manylinux1_x86_64.whl (890.2 MB)\n",
-      "Collecting torchaudio\n",
-      "  Using cached torchaudio-0.13.0-cp38-cp38-manylinux1_x86_64.whl (4.2 MB)\n",
-      "Collecting torchvision\n",
-      "  Downloading torchvision-0.14.0-cp38-cp38-manylinux1_x86_64.whl (24.3 MB)\n",
-      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.3/24.3 MB\u001b[0m \u001b[31m103.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
-      "\u001b[?25hCollecting nvidia-cuda-runtime-cu11==11.7.99\n",
-      "  Using cached nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
-      "Collecting typing-extensions\n",
-      "  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)\n",
-      "Collecting nvidia-cuda-nvrtc-cu11==11.7.99\n",
-      "  Using cached nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
-      "Collecting nvidia-cudnn-cu11==8.5.0.96\n",
-      "  Using cached nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
-      "Collecting nvidia-cublas-cu11==11.10.3.66\n",
-      "  Using cached nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
-      "Collecting setuptools\n",
-      "  Using cached setuptools-65.6.3-py3-none-any.whl (1.2 MB)\n",
-      "Collecting wheel\n",
-      "  Using cached wheel-0.38.4-py3-none-any.whl (36 kB)\n",
-      "Collecting numpy\n",
-      "  Downloading numpy-1.24.0rc2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n",
-      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.3/17.3 MB\u001b[0m \u001b[31m100.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
-      "\u001b[?25hCollecting pillow!=8.3.*,>=5.3.0\n",
-      "  Downloading Pillow-9.3.0-cp38-cp38-manylinux_2_28_x86_64.whl (3.3 MB)\n",
-      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.3/3.3 MB\u001b[0m \u001b[31m145.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
-      "\u001b[?25hCollecting requests\n",
-      "  Using cached requests-2.28.1-py3-none-any.whl (62 kB)\n",
-      "Collecting certifi>=2017.4.17\n",
-      "  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)\n",
-      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m155.3/155.3 kB\u001b[0m \u001b[31m44.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
-      "\u001b[?25hCollecting charset-normalizer<3,>=2\n",
-      "  Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)\n",
-      "Collecting urllib3<1.27,>=1.21.1\n",
-      "  Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB)\n",
-      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m140.6/140.6 kB\u001b[0m \u001b[31m38.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
-      "\u001b[?25hCollecting idna<4,>=2.5\n",
-      "  Downloading idna-3.4-py3-none-any.whl (61 kB)\n",
-      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.5/61.5 kB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
-      "\u001b[?25hInstalling collected packages: wheel, urllib3, typing-extensions, setuptools, pillow, nvidia-cuda-nvrtc-cu11, numpy, idna, charset-normalizer, certifi, requests, nvidia-cuda-runtime-cu11, nvidia-cublas-cu11, nvidia-cudnn-cu11, torch, torchvision, torchaudio\n",
-      "  Attempting uninstall: typing-extensions\n",
-      "    Found existing installation: typing_extensions 4.4.0\n",
-      "    Uninstalling typing_extensions-4.4.0:\n",
-      "      Successfully uninstalled typing_extensions-4.4.0\n",
-      "  Attempting uninstall: nvidia-cuda-nvrtc-cu11\n",
-      "    Found existing installation: nvidia-cuda-nvrtc-cu11 11.7.99\n",
-      "    Uninstalling nvidia-cuda-nvrtc-cu11-11.7.99:\n",
-      "      Successfully uninstalled nvidia-cuda-nvrtc-cu11-11.7.99\n",
-      "  Attempting uninstall: numpy\n",
-      "    Found existing installation: numpy 1.22.4\n",
-      "    Uninstalling numpy-1.22.4:\n",
-      "      Successfully uninstalled numpy-1.22.4\n",
-      "  Attempting uninstall: charset-normalizer\n",
-      "    Found existing installation: charset-normalizer 2.1.1\n",
-      "    Uninstalling charset-normalizer-2.1.1:\n",
-      "      Successfully uninstalled charset-normalizer-2.1.1\n",
-      "  Attempting uninstall: requests\n",
-      "    Found existing installation: requests 2.28.1\n",
-      "    Uninstalling requests-2.28.1:\n",
-      "      Successfully uninstalled requests-2.28.1\n",
-      "  Attempting uninstall: nvidia-cuda-runtime-cu11\n",
-      "    Found existing installation: nvidia-cuda-runtime-cu11 11.7.99\n",
-      "    Uninstalling nvidia-cuda-runtime-cu11-11.7.99:\n",
-      "      Successfully uninstalled nvidia-cuda-runtime-cu11-11.7.99\n",
-      "  Attempting uninstall: nvidia-cublas-cu11\n",
-      "    Found existing installation: nvidia-cublas-cu11 11.10.3.66\n",
-      "    Uninstalling nvidia-cublas-cu11-11.10.3.66:\n",
-      "      Successfully uninstalled nvidia-cublas-cu11-11.10.3.66\n",
-      "  Attempting uninstall: nvidia-cudnn-cu11\n",
-      "    Found existing installation: nvidia-cudnn-cu11 8.5.0.96\n",
-      "    Uninstalling nvidia-cudnn-cu11-8.5.0.96:\n",
-      "      Successfully uninstalled nvidia-cudnn-cu11-8.5.0.96\n",
-      "  Attempting uninstall: torch\n",
-      "    Found existing installation: torch 1.13.0\n",
-      "    Uninstalling torch-1.13.0:\n",
-      "      Successfully uninstalled torch-1.13.0\n",
-      "  Attempting uninstall: torchaudio\n",
-      "    Found existing installation: torchaudio 0.13.0\n",
-      "    Uninstalling torchaudio-0.13.0:\n",
-      "      Successfully uninstalled torchaudio-0.13.0\n",
-      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
-      "launchpadlib 1.10.13 requires testresources, which is not installed.\n",
-      "pandas-profiling 3.4.0 requires numpy<1.24,>=1.16.0, but you have numpy 1.24.0rc2 which is incompatible.\n",
-      "numba 0.56.4 requires numpy<1.24,>=1.18, but you have numpy 1.24.0rc2 which is incompatible.\u001b[0m\u001b[31m\n",
-      "\u001b[0mSuccessfully installed certifi-2022.12.7 charset-normalizer-2.1.1 idna-3.4 numpy-1.24.0rc2 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 pillow-9.3.0 requests-2.28.1 setuptools-65.6.3 torch-1.13.0 torchaudio-0.13.0 torchvision-0.14.0 typing-extensions-4.4.0 urllib3-1.26.13 wheel-0.38.4\n"
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mRequirement already satisfied: pip in ./.local/lib/python3.8/site-packages (22.3.1)\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mDefaulting to user installation because normal site-packages is not writeable\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mRequirement already satisfied: torch in ./.local/lib/python3.8/site-packages (1.13.0)\n",
+      "Requirement already satisfied: torchaudio in ./.local/lib/python3.8/site-packages (0.13.0)\n",
+      "Requirement already satisfied: torchvision in ./.local/lib/python3.8/site-packages (0.14.0)\n",
+      "Requirement already satisfied: typing-extensions in ./.local/lib/python3.8/site-packages (from torch) (4.4.0)\n",
+      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in ./.local/lib/python3.8/site-packages (from torch) (8.5.0.96)\n",
+      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in ./.local/lib/python3.8/site-packages (from torch) (11.10.3.66)\n",
+      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in ./.local/lib/python3.8/site-packages (from torch) (11.7.99)\n",
+      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in ./.local/lib/python3.8/site-packages (from torch) (11.7.99)\n",
+      "Requirement already satisfied: setuptools in ./.local/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch) (65.6.3)\n",
+      "Requirement already satisfied: wheel in ./.local/lib/python3.8/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch) (0.38.4)\n",
+      "Requirement already satisfied: numpy in ./.local/lib/python3.8/site-packages (from torchvision) (1.23.5)\n",
+      "Requirement already satisfied: requests in ./.local/lib/python3.8/site-packages (from torchvision) (2.28.1)\n",
+      "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./.local/lib/python3.8/site-packages (from torchvision) (9.3.0)\n",
+      "Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.8/site-packages (from requests->torchvision) (3.4)\n",
+      "Requirement already satisfied: certifi>=2017.4.17 in ./.local/lib/python3.8/site-packages (from requests->torchvision) (2022.12.7)\n",
+      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.local/lib/python3.8/site-packages (from requests->torchvision) (1.26.13)\n",
+      "Requirement already satisfied: charset-normalizer<3,>=2 in ./.local/lib/python3.8/site-packages (from requests->torchvision) (2.1.1)\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m"
      ]
     }
    ],
    "source": [
     "!pip3 install --upgrade pip\n",
-    "!pip3 install \"numpy<1.23.0\"\n",
+    "!pip3 install --upgrade numpy>=1.18\n",
+    "!pip3 install --upgrade packaging>=20.9\n",
+    "!pip3 install --upgrade typing-extensions>=3.7.4.3\n",
     "\n",
-    "!pip3 install --pre torch torchaudio torchvision --force-reinstall\n",
+    "!pip3 install --pre torch torchaudio torchvision --upgrade\n",
     "\n",
-    "!pip3 install bitsandbytes\n",
+    "#!pip3 install bitsandbytes\n",
     "\n",
     "\n",
     "#!pip3 install --pre torch torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cu116\n",
@@ -269,7 +231,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Thu Dec  8 18:45:37 2022       \n",
+      "Mon Dec 12 01:24:00 2022       \n",
       "+-----------------------------------------------------------------------------+\n",
       "| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |\n",
       "|-------------------------------+----------------------+----------------------+\n",
@@ -278,7 +240,7 @@
       "|                               |                      |               MIG M. |\n",
       "|===============================+======================+======================|\n",
       "|   0  NVIDIA A100-SXM...  On   | 00000000:06:00.0 Off |                    0 |\n",
-      "| N/A   31C    P0    47W / 400W |      0MiB / 40960MiB |      0%      Default |\n",
+      "| N/A   33C    P0    44W / 400W |      0MiB / 40960MiB |      0%      Default |\n",
       "|                               |                      |             Disabled |\n",
       "+-------------------------------+----------------------+----------------------+\n",
       "                                                                               \n",
@@ -330,40 +292,40 @@
      "output_type": "stream",
      "text": [
       "Get:1 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease [1484 B]\n",
-      "Hit:2 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease\n",
-      "Hit:3 https://download.docker.com/linux/ubuntu focal InRelease                 \n",
-      "Hit:4 http://archive.lambdalabs.com/ubuntu focal InRelease                     \n",
-      "Hit:5 https://packages.cloud.google.com/apt cloud-sdk InRelease                \n",
-      "Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease               \n",
-      "Ign:7 http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal InRelease       \n",
-      "Hit:8 http://archive.ubuntu.com/ubuntu focal InRelease                         \n",
-      "Hit:9 https://packages.microsoft.com/repos/azure-cli focal InRelease           \n",
-      "Hit:10 http://archive.ubuntu.com/ubuntu focal-updates InRelease                \n",
-      "Hit:11 https://pkg.cloudflare.com/cloudflared focal InRelease                  \n",
-      "Hit:12 http://archive.ubuntu.com/ubuntu focal-backports InRelease              \n",
-      "Err:13 http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal Release        \n",
+      "Hit:2 https://download.docker.com/linux/ubuntu focal InRelease                 \n",
+      "Hit:3 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease\n",
+      "Hit:4 https://packages.cloud.google.com/apt cloud-sdk InRelease                \n",
+      "Hit:5 https://packages.microsoft.com/repos/azure-cli focal InRelease           \n",
+      "Ign:6 http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal InRelease       \n",
+      "Hit:7 http://archive.lambdalabs.com/ubuntu focal InRelease                     \n",
+      "Hit:8 https://pkg.cloudflare.com/cloudflared focal InRelease                   \n",
+      "Err:9 http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal Release         \n",
       "  404  Not Found [IP: 185.125.190.52 80]\n",
-      "Hit:14 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu focal InRelease  \n",
-      "Reading package lists... Done\n",
+      "Hit:10 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu focal InRelease  \n",
+      "Hit:11 http://archive.ubuntu.com/ubuntu focal InRelease                        \n",
+      "Hit:12 http://archive.ubuntu.com/ubuntu focal-updates InRelease\n",
+      "Get:13 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]\n",
+      "Get:14 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]\n",
+      "Reading package lists... Done                                    \n",
       "E: The repository 'http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal Release' does not have a Release file.\n",
       "N: Updating from such a repository can't be done securely, and is therefore disabled by default.\n",
       "N: See apt-secure(8) manpage for repository creation and user configuration details.\n",
-      "Get:1 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease [1484 B]\n",
-      "Hit:2 https://download.docker.com/linux/ubuntu focal InRelease                 \u001b[0m\u001b[33m\n",
+      "Hit:1 https://download.docker.com/linux/ubuntu focal InRelease\n",
+      "Get:2 https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease [1484 B]\n",
       "Hit:3 https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/amd64  InRelease\n",
-      "Hit:4 https://packages.cloud.google.com/apt cloud-sdk InRelease                \u001b[0m\u001b[33m\u001b[33m\n",
-      "Hit:5 http://archive.lambdalabs.com/ubuntu focal InRelease                     \u001b[0m\n",
-      "Hit:6 http://archive.ubuntu.com/ubuntu focal InRelease                         \u001b[0m\u001b[33m\n",
-      "Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease               \u001b[0m\n",
-      "Ign:8 http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal InRelease       \u001b[0m\n",
-      "Hit:9 https://packages.microsoft.com/repos/azure-cli focal InRelease           \u001b[0m\u001b[33m\n",
-      "Hit:10 http://archive.ubuntu.com/ubuntu focal-updates InRelease                \u001b[0m\n",
-      "Hit:11 https://pkg.cloudflare.com/cloudflared focal InRelease                  \u001b[0m\u001b[33m\n",
-      "Hit:12 http://archive.ubuntu.com/ubuntu focal-backports InRelease   \u001b[0m       \u001b[0m\u001b[33m\n",
-      "Hit:13 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu focal InRelease\n",
-      "Err:14 http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal Release\n",
+      "Hit:4 https://packages.microsoft.com/repos/azure-cli focal InRelease           \u001b[0m\n",
+      "Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]      \u001b[0m\n",
+      "Hit:6 https://packages.cloud.google.com/apt cloud-sdk InRelease                \u001b[0m\n",
+      "Hit:7 http://archive.ubuntu.com/ubuntu focal InRelease                         \u001b[0m\u001b[33m\u001b[33m\n",
+      "Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease                 \u001b[0m\n",
+      "Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]      \u001b[0m\n",
+      "Ign:10 http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal InRelease      \u001b[0m\u001b[33m\n",
+      "Hit:11 http://archive.lambdalabs.com/ubuntu focal InRelease                    \u001b[0m\u001b[33m\u001b[33m\n",
+      "Hit:12 https://pkg.cloudflare.com/cloudflared focal InRelease                  \u001b[0m\u001b[33m\n",
+      "Err:13 http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal Release        \u001b[0m\u001b[33m\n",
       "  404  Not Found [IP: 185.125.190.52 80]\n",
-      "Reading package lists... Done\u001b[33m\n",
+      "Hit:14 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu focal InRelease\n",
+      "Reading package lists... Done\u001b[33m\u001b[33m\u001b[33m\n",
       "\u001b[1;31mE: \u001b[0mThe repository 'http://ppa.launchpad.net/jonathonf/ffmpeg-4/ubuntu focal Release' does not have a Release file.\u001b[0m\n",
       "\u001b[33mN: \u001b[0mUpdating from such a repository can't be done securely, and is therefore disabled by default.\u001b[0m\n",
       "\u001b[33mN: \u001b[0mSee apt-secure(8) manpage for repository creation and user configuration details.\u001b[0m\n",
@@ -413,172 +375,207 @@
      "output_type": "stream",
      "text": [
       "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Collecting git+https://github.com/huggingface/datasets\n",
-      "  Cloning https://github.com/huggingface/datasets to /tmp/pip-req-build-_aqg2yxr\n",
-      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/datasets /tmp/pip-req-build-_aqg2yxr\n",
-      "  Resolved https://github.com/huggingface/datasets to commit 45508f7d8858579c62d93779873ef5eb6b05bc74\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mCollecting git+https://github.com/huggingface/datasets\n",
+      "  Cloning https://github.com/huggingface/datasets to /tmp/pip-req-build-fzgi4uj2\n",
+      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/datasets /tmp/pip-req-build-fzgi4uj2\n",
+      "  Resolved https://github.com/huggingface/datasets to commit 5266c81430628edc175013692f02f5f2747ff29e\n",
       "  Installing build dependencies ... \u001b[?25ldone\n",
       "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
       "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
-      "\u001b[?25hRequirement already satisfied: dill<0.3.7 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (0.3.6)\n",
+      "\u001b[?25hRequirement already satisfied: xxhash in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (3.1.0)\n",
+      "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (2022.11.0)\n",
+      "Requirement already satisfied: numpy>=1.17 in ./.local/lib/python3.8/site-packages (from datasets==2.7.1.dev0) (1.23.5)\n",
+      "Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (0.11.1)\n",
+      "Requirement already satisfied: tqdm>=4.62.1 in ./.local/lib/python3.8/site-packages (from datasets==2.7.1.dev0) (4.64.1)\n",
+      "Requirement already satisfied: dill<0.3.7 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (0.3.6)\n",
+      "Requirement already satisfied: packaging in ./.local/lib/python3.8/site-packages (from datasets==2.7.1.dev0) (22.0)\n",
       "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (10.0.1)\n",
+      "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (3.8.3)\n",
       "Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3/dist-packages (from datasets==2.7.1.dev0) (5.3.1)\n",
       "Requirement already satisfied: pandas in ./.local/lib/python3.8/site-packages (from datasets==2.7.1.dev0) (1.5.1)\n",
-      "Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (0.11.1)\n",
-      "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (3.8.3)\n",
-      "Requirement already satisfied: numpy>=1.17 in ./.local/lib/python3.8/site-packages (from datasets==2.7.1.dev0) (1.24.0rc2)\n",
-      "Requirement already satisfied: multiprocess in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (0.70.14)\n",
-      "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (2022.11.0)\n",
-      "Requirement already satisfied: packaging in ./.local/lib/python3.8/site-packages (from datasets==2.7.1.dev0) (21.3)\n",
-      "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (0.18.0)\n",
       "Requirement already satisfied: requests>=2.19.0 in ./.local/lib/python3.8/site-packages (from datasets==2.7.1.dev0) (2.28.1)\n",
-      "Requirement already satisfied: tqdm>=4.62.1 in ./.local/lib/python3.8/site-packages (from datasets==2.7.1.dev0) (4.64.1)\n",
-      "Requirement already satisfied: xxhash in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (3.1.0)\n",
+      "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (0.18.0)\n",
+      "Requirement already satisfied: multiprocess in /usr/local/lib/python3.8/dist-packages (from datasets==2.7.1.dev0) (0.70.14)\n",
       "Requirement already satisfied: attrs>=17.3.0 in /usr/lib/python3/dist-packages (from aiohttp->datasets==2.7.1.dev0) (19.3.0)\n",
+      "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets==2.7.1.dev0) (6.0.3)\n",
       "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets==2.7.1.dev0) (1.3.3)\n",
-      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets==2.7.1.dev0) (4.0.2)\n",
       "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in ./.local/lib/python3.8/site-packages (from aiohttp->datasets==2.7.1.dev0) (2.1.1)\n",
+      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets==2.7.1.dev0) (4.0.2)\n",
       "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets==2.7.1.dev0) (1.8.2)\n",
-      "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets==2.7.1.dev0) (6.0.3)\n",
       "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets==2.7.1.dev0) (1.3.1)\n",
-      "Requirement already satisfied: filelock in /usr/lib/python3/dist-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets==2.7.1.dev0) (3.0.12)\n",
       "Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.local/lib/python3.8/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets==2.7.1.dev0) (4.4.0)\n",
-      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3/dist-packages (from packaging->datasets==2.7.1.dev0) (2.4.6)\n",
+      "Requirement already satisfied: filelock in /usr/lib/python3/dist-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets==2.7.1.dev0) (3.0.12)\n",
+      "Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->datasets==2.7.1.dev0) (3.4)\n",
       "Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->datasets==2.7.1.dev0) (1.26.13)\n",
       "Requirement already satisfied: certifi>=2017.4.17 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->datasets==2.7.1.dev0) (2022.12.7)\n",
-      "Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->datasets==2.7.1.dev0) (3.4)\n",
       "Requirement already satisfied: python-dateutil>=2.8.1 in ./.local/lib/python3.8/site-packages (from pandas->datasets==2.7.1.dev0) (2.8.2)\n",
       "Requirement already satisfied: pytz>=2020.1 in ./.local/lib/python3.8/site-packages (from pandas->datasets==2.7.1.dev0) (2022.5)\n",
       "Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.1->pandas->datasets==2.7.1.dev0) (1.14.0)\n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Collecting git+https://github.com/huggingface/transformers\n",
-      "  Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-x539p2ep\n",
-      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers /tmp/pip-req-build-x539p2ep\n",
-      "  Resolved https://github.com/huggingface/transformers to commit e3cc4487fe66e03ec85970ea2db8e5fb34c455f4\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mDefaulting to user installation because normal site-packages is not writeable\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mCollecting git+https://github.com/huggingface/transformers\n",
+      "  Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-3doeohfs\n",
+      "  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers /tmp/pip-req-build-3doeohfs\n",
+      "  Resolved https://github.com/huggingface/transformers to commit 799cea64ac1029d66e9e58f18bc6f47892270723\n",
       "  Installing build dependencies ... \u001b[?25ldone\n",
       "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
       "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
-      "\u001b[?25hRequirement already satisfied: huggingface-hub<1.0,>=0.10.0 in /usr/local/lib/python3.8/dist-packages (from transformers==4.26.0.dev0) (0.11.1)\n",
+      "\u001b[?25hRequirement already satisfied: pyyaml>=5.1 in /usr/lib/python3/dist-packages (from transformers==4.26.0.dev0) (5.3.1)\n",
+      "Requirement already satisfied: packaging>=20.0 in ./.local/lib/python3.8/site-packages (from transformers==4.26.0.dev0) (22.0)\n",
+      "Requirement already satisfied: tqdm>=4.27 in ./.local/lib/python3.8/site-packages (from transformers==4.26.0.dev0) (4.64.1)\n",
+      "Requirement already satisfied: huggingface-hub<1.0,>=0.10.0 in /usr/local/lib/python3.8/dist-packages (from transformers==4.26.0.dev0) (0.11.1)\n",
       "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers==4.26.0.dev0) (2022.10.31)\n",
-      "Requirement already satisfied: numpy>=1.17 in ./.local/lib/python3.8/site-packages (from transformers==4.26.0.dev0) (1.24.0rc2)\n",
-      "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers==4.26.0.dev0) (0.13.2)\n",
-      "Requirement already satisfied: filelock in /usr/lib/python3/dist-packages (from transformers==4.26.0.dev0) (3.0.12)\n",
       "Requirement already satisfied: requests in ./.local/lib/python3.8/site-packages (from transformers==4.26.0.dev0) (2.28.1)\n",
-      "Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3/dist-packages (from transformers==4.26.0.dev0) (5.3.1)\n",
-      "Requirement already satisfied: packaging>=20.0 in ./.local/lib/python3.8/site-packages (from transformers==4.26.0.dev0) (21.3)\n",
-      "Requirement already satisfied: tqdm>=4.27 in ./.local/lib/python3.8/site-packages (from transformers==4.26.0.dev0) (4.64.1)\n",
+      "Requirement already satisfied: filelock in /usr/lib/python3/dist-packages (from transformers==4.26.0.dev0) (3.0.12)\n",
+      "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers==4.26.0.dev0) (0.13.2)\n",
+      "Requirement already satisfied: numpy>=1.17 in ./.local/lib/python3.8/site-packages (from transformers==4.26.0.dev0) (1.23.5)\n",
       "Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.local/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.10.0->transformers==4.26.0.dev0) (4.4.0)\n",
-      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3/dist-packages (from packaging>=20.0->transformers==4.26.0.dev0) (2.4.6)\n",
-      "Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.8/site-packages (from requests->transformers==4.26.0.dev0) (3.4)\n",
-      "Requirement already satisfied: charset-normalizer<3,>=2 in ./.local/lib/python3.8/site-packages (from requests->transformers==4.26.0.dev0) (2.1.1)\n",
       "Requirement already satisfied: certifi>=2017.4.17 in ./.local/lib/python3.8/site-packages (from requests->transformers==4.26.0.dev0) (2022.12.7)\n",
+      "Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.8/site-packages (from requests->transformers==4.26.0.dev0) (3.4)\n",
       "Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.local/lib/python3.8/site-packages (from requests->transformers==4.26.0.dev0) (1.26.13)\n",
-      "/usr/bin/sh: 1: cannot create =2.7.1: Permission denied\n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Requirement already satisfied: librosa in /usr/local/lib/python3.8/dist-packages (0.9.2)\n",
-      "Requirement already satisfied: scikit-learn>=0.19.1 in /usr/lib/python3/dist-packages (from librosa) (0.22.2.post1)\n",
-      "Requirement already satisfied: audioread>=2.1.9 in /usr/local/lib/python3.8/dist-packages (from librosa) (3.0.0)\n",
-      "Requirement already satisfied: joblib>=0.14 in ./.local/lib/python3.8/site-packages (from librosa) (1.2.0)\n",
+      "Requirement already satisfied: charset-normalizer<3,>=2 in ./.local/lib/python3.8/site-packages (from requests->transformers==4.26.0.dev0) (2.1.1)\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mDefaulting to user installation because normal site-packages is not writeable\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mRequirement already satisfied: librosa in /usr/local/lib/python3.8/dist-packages (0.9.2)\n",
       "Requirement already satisfied: decorator>=4.0.10 in /usr/lib/python3/dist-packages (from librosa) (4.4.2)\n",
-      "Requirement already satisfied: soundfile>=0.10.2 in /usr/local/lib/python3.8/dist-packages (from librosa) (0.11.0)\n",
       "Requirement already satisfied: pooch>=1.0 in /usr/local/lib/python3.8/dist-packages (from librosa) (1.6.0)\n",
-      "Requirement already satisfied: packaging>=20.0 in ./.local/lib/python3.8/site-packages (from librosa) (21.3)\n",
-      "Requirement already satisfied: numpy>=1.17.0 in ./.local/lib/python3.8/site-packages (from librosa) (1.24.0rc2)\n",
+      "Requirement already satisfied: numba>=0.45.1 in /usr/local/lib/python3.8/dist-packages (from librosa) (0.56.4)\n",
+      "Requirement already satisfied: scikit-learn>=0.19.1 in /usr/lib/python3/dist-packages (from librosa) (0.22.2.post1)\n",
       "Requirement already satisfied: scipy>=1.2.0 in ./.local/lib/python3.8/site-packages (from librosa) (1.9.3)\n",
+      "Requirement already satisfied: packaging>=20.0 in ./.local/lib/python3.8/site-packages (from librosa) (22.0)\n",
+      "Requirement already satisfied: soundfile>=0.10.2 in /usr/local/lib/python3.8/dist-packages (from librosa) (0.11.0)\n",
+      "Requirement already satisfied: numpy>=1.17.0 in ./.local/lib/python3.8/site-packages (from librosa) (1.23.5)\n",
+      "Requirement already satisfied: audioread>=2.1.9 in /usr/local/lib/python3.8/dist-packages (from librosa) (3.0.0)\n",
+      "Requirement already satisfied: joblib>=0.14 in ./.local/lib/python3.8/site-packages (from librosa) (1.2.0)\n",
       "Requirement already satisfied: resampy>=0.2.2 in /usr/local/lib/python3.8/dist-packages (from librosa) (0.4.2)\n",
-      "Requirement already satisfied: numba>=0.45.1 in /usr/local/lib/python3.8/dist-packages (from librosa) (0.56.4)\n",
-      "Collecting numpy>=1.17.0\n",
-      "  Downloading numpy-1.23.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)\n",
-      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m17.1/17.1 MB\u001b[0m \u001b[31m104.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
-      "\u001b[?25hRequirement already satisfied: setuptools in ./.local/lib/python3.8/site-packages (from numba>=0.45.1->librosa) (65.6.3)\n",
       "Requirement already satisfied: importlib-metadata in ./.local/lib/python3.8/site-packages (from numba>=0.45.1->librosa) (5.0.0)\n",
       "Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /usr/local/lib/python3.8/dist-packages (from numba>=0.45.1->librosa) (0.39.1)\n",
-      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3/dist-packages (from packaging>=20.0->librosa) (2.4.6)\n",
+      "Requirement already satisfied: setuptools in ./.local/lib/python3.8/site-packages (from numba>=0.45.1->librosa) (65.6.3)\n",
       "Requirement already satisfied: requests>=2.19.0 in ./.local/lib/python3.8/site-packages (from pooch>=1.0->librosa) (2.28.1)\n",
       "Requirement already satisfied: appdirs>=1.3.0 in /usr/lib/python3/dist-packages (from pooch>=1.0->librosa) (1.4.3)\n",
       "Requirement already satisfied: cffi>=1.0 in /usr/lib/python3/dist-packages (from soundfile>=0.10.2->librosa) (1.14.0)\n",
       "Requirement already satisfied: certifi>=2017.4.17 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->pooch>=1.0->librosa) (2022.12.7)\n",
+      "Requirement already satisfied: charset-normalizer<3,>=2 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->pooch>=1.0->librosa) (2.1.1)\n",
       "Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->pooch>=1.0->librosa) (3.4)\n",
       "Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->pooch>=1.0->librosa) (1.26.13)\n",
-      "Requirement already satisfied: charset-normalizer<3,>=2 in ./.local/lib/python3.8/site-packages (from requests>=2.19.0->pooch>=1.0->librosa) (2.1.1)\n",
       "Requirement already satisfied: zipp>=0.5 in /usr/lib/python3/dist-packages (from importlib-metadata->numba>=0.45.1->librosa) (1.0.0)\n",
-      "Installing collected packages: numpy\n",
-      "  Attempting uninstall: numpy\n",
-      "    Found existing installation: numpy 1.24.0rc2\n",
-      "    Uninstalling numpy-1.24.0rc2:\n",
-      "      Successfully uninstalled numpy-1.24.0rc2\n",
-      "Successfully installed numpy-1.23.5\n",
-      "/usr/bin/sh: 1: cannot create =0.3.0: Permission denied\n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Requirement already satisfied: jiwer in /usr/local/lib/python3.8/dist-packages (2.5.1)\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mDefaulting to user installation because normal site-packages is not writeable\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mRequirement already satisfied: jiwer in /usr/local/lib/python3.8/dist-packages (2.5.1)\n",
       "Requirement already satisfied: levenshtein==0.20.2 in /usr/local/lib/python3.8/dist-packages (from jiwer) (0.20.2)\n",
-      "Requirement already satisfied: rapidfuzz<3.0.0,>=2.3.0 in /usr/local/lib/python3.8/dist-packages (from levenshtein==0.20.2->jiwer) (2.13.3)\n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Requirement already satisfied: gradio in /usr/local/lib/python3.8/dist-packages (3.12.0)\n",
-      "Requirement already satisfied: pillow in ./.local/lib/python3.8/site-packages (from gradio) (9.3.0)\n",
-      "Requirement already satisfied: requests in ./.local/lib/python3.8/site-packages (from gradio) (2.28.1)\n",
+      "Requirement already satisfied: rapidfuzz<3.0.0,>=2.3.0 in /usr/local/lib/python3.8/dist-packages (from levenshtein==0.20.2->jiwer) (2.13.5)\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mDefaulting to user installation because normal site-packages is not writeable\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mRequirement already satisfied: gradio in ./.local/lib/python3.8/site-packages (3.12.0)\n",
+      "Requirement already satisfied: matplotlib in ./.local/lib/python3.8/site-packages (from gradio) (3.5.3)\n",
+      "Requirement already satisfied: fastapi in ./.local/lib/python3.8/site-packages (from gradio) (0.88.0)\n",
+      "Requirement already satisfied: h11<0.13,>=0.11 in ./.local/lib/python3.8/site-packages (from gradio) (0.12.0)\n",
+      "Requirement already satisfied: orjson in ./.local/lib/python3.8/site-packages (from gradio) (3.8.3)\n",
+      "Requirement already satisfied: httpx in ./.local/lib/python3.8/site-packages (from gradio) (0.23.1)\n",
+      "Requirement already satisfied: pycryptodome in ./.local/lib/python3.8/site-packages (from gradio) (3.16.0)\n",
       "Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from gradio) (5.3.1)\n",
-      "Requirement already satisfied: pycryptodome in /usr/local/lib/python3.8/dist-packages (from gradio) (3.16.0)\n",
-      "Requirement already satisfied: ffmpy in /usr/local/lib/python3.8/dist-packages (from gradio) (0.3.0)\n",
-      "Requirement already satisfied: websockets>=10.0 in /usr/local/lib/python3.8/dist-packages (from gradio) (10.4)\n",
+      "Requirement already satisfied: websockets>=10.0 in ./.local/lib/python3.8/site-packages (from gradio) (10.4)\n",
+      "Requirement already satisfied: requests in ./.local/lib/python3.8/site-packages (from gradio) (2.28.1)\n",
+      "Requirement already satisfied: ffmpy in ./.local/lib/python3.8/site-packages (from gradio) (0.3.0)\n",
+      "Requirement already satisfied: numpy in ./.local/lib/python3.8/site-packages (from gradio) (1.23.5)\n",
       "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from gradio) (3.8.3)\n",
-      "Requirement already satisfied: paramiko in /usr/local/lib/python3.8/dist-packages (from gradio) (2.12.0)\n",
-      "Requirement already satisfied: pydub in /usr/local/lib/python3.8/dist-packages (from gradio) (0.25.1)\n",
-      "Requirement already satisfied: fsspec in /usr/local/lib/python3.8/dist-packages (from gradio) (2022.11.0)\n",
-      "Requirement already satisfied: fastapi in /usr/local/lib/python3.8/dist-packages (from gradio) (0.88.0)\n",
-      "Requirement already satisfied: python-multipart in /usr/local/lib/python3.8/dist-packages (from gradio) (0.0.5)\n",
+      "Requirement already satisfied: paramiko in ./.local/lib/python3.8/site-packages (from gradio) (2.12.0)\n",
+      "Requirement already satisfied: pandas in ./.local/lib/python3.8/site-packages (from gradio) (1.5.1)\n",
+      "Requirement already satisfied: pydub in ./.local/lib/python3.8/site-packages (from gradio) (0.25.1)\n",
+      "Requirement already satisfied: pillow in ./.local/lib/python3.8/site-packages (from gradio) (9.3.0)\n",
+      "Requirement already satisfied: uvicorn in ./.local/lib/python3.8/site-packages (from gradio) (0.20.0)\n",
       "Requirement already satisfied: pydantic in ./.local/lib/python3.8/site-packages (from gradio) (1.10.2)\n",
-      "Requirement already satisfied: orjson in /usr/local/lib/python3.8/dist-packages (from gradio) (3.8.3)\n",
-      "Requirement already satisfied: h11<0.13,>=0.11 in /usr/local/lib/python3.8/dist-packages (from gradio) (0.12.0)\n",
-      "Requirement already satisfied: matplotlib in ./.local/lib/python3.8/site-packages (from gradio) (3.5.3)\n",
-      "Requirement already satisfied: markdown-it-py[linkify,plugins] in /usr/local/lib/python3.8/dist-packages (from gradio) (2.1.0)\n",
+      "Requirement already satisfied: markdown-it-py[linkify,plugins] in ./.local/lib/python3.8/site-packages (from gradio) (2.1.0)\n",
+      "Requirement already satisfied: python-multipart in ./.local/lib/python3.8/site-packages (from gradio) (0.0.5)\n",
+      "Requirement already satisfied: fsspec in /usr/local/lib/python3.8/dist-packages (from gradio) (2022.11.0)\n",
       "Requirement already satisfied: jinja2 in ./.local/lib/python3.8/site-packages (from gradio) (3.1.2)\n",
-      "Requirement already satisfied: numpy in ./.local/lib/python3.8/site-packages (from gradio) (1.23.5)\n",
-      "Requirement already satisfied: pandas in ./.local/lib/python3.8/site-packages (from gradio) (1.5.1)\n",
-      "Requirement already satisfied: uvicorn in /usr/local/lib/python3.8/dist-packages (from gradio) (0.20.0)\n",
-      "Requirement already satisfied: httpx in /usr/local/lib/python3.8/dist-packages (from gradio) (0.23.1)\n",
       "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->gradio) (1.8.2)\n",
+      "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in ./.local/lib/python3.8/site-packages (from aiohttp->gradio) (2.1.1)\n",
       "Requirement already satisfied: attrs>=17.3.0 in /usr/lib/python3/dist-packages (from aiohttp->gradio) (19.3.0)\n",
-      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->gradio) (4.0.2)\n",
+      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->gradio) (1.3.1)\n",
       "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->gradio) (6.0.3)\n",
       "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->gradio) (1.3.3)\n",
-      "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in ./.local/lib/python3.8/site-packages (from aiohttp->gradio) (2.1.1)\n",
-      "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->gradio) (1.3.1)\n",
-      "Requirement already satisfied: starlette==0.22.0 in /usr/local/lib/python3.8/dist-packages (from fastapi->gradio) (0.22.0)\n",
+      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->gradio) (4.0.2)\n",
+      "Requirement already satisfied: starlette==0.22.0 in ./.local/lib/python3.8/site-packages (from fastapi->gradio) (0.22.0)\n",
       "Requirement already satisfied: typing-extensions>=3.10.0 in ./.local/lib/python3.8/site-packages (from starlette==0.22.0->fastapi->gradio) (4.4.0)\n",
       "Requirement already satisfied: anyio<5,>=3.4.0 in ./.local/lib/python3.8/site-packages (from starlette==0.22.0->fastapi->gradio) (3.6.2)\n",
-      "Requirement already satisfied: httpcore<0.17.0,>=0.15.0 in /usr/local/lib/python3.8/dist-packages (from httpx->gradio) (0.15.0)\n",
-      "Requirement already satisfied: sniffio in ./.local/lib/python3.8/site-packages (from httpx->gradio) (1.3.0)\n",
-      "Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in /usr/local/lib/python3.8/dist-packages (from httpx->gradio) (1.5.0)\n",
+      "Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in ./.local/lib/python3.8/site-packages (from httpx->gradio) (1.5.0)\n",
+      "Requirement already satisfied: httpcore<0.17.0,>=0.15.0 in ./.local/lib/python3.8/site-packages (from httpx->gradio) (0.15.0)\n",
       "Requirement already satisfied: certifi in ./.local/lib/python3.8/site-packages (from httpx->gradio) (2022.12.7)\n",
+      "Requirement already satisfied: sniffio in ./.local/lib/python3.8/site-packages (from httpx->gradio) (1.3.0)\n",
       "Requirement already satisfied: MarkupSafe>=2.0 in ./.local/lib/python3.8/site-packages (from jinja2->gradio) (2.1.1)\n",
-      "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.8/dist-packages (from markdown-it-py[linkify,plugins]->gradio) (0.1.2)\n",
-      "Requirement already satisfied: mdit-py-plugins in /usr/local/lib/python3.8/dist-packages (from markdown-it-py[linkify,plugins]->gradio) (0.3.3)\n",
-      "Requirement already satisfied: linkify-it-py~=1.0 in /usr/local/lib/python3.8/dist-packages (from markdown-it-py[linkify,plugins]->gradio) (1.0.3)\n",
-      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/lib/python3/dist-packages (from matplotlib->gradio) (1.0.1)\n",
+      "Requirement already satisfied: mdurl~=0.1 in ./.local/lib/python3.8/site-packages (from markdown-it-py[linkify,plugins]->gradio) (0.1.2)\n",
+      "Requirement already satisfied: mdit-py-plugins in ./.local/lib/python3.8/site-packages (from markdown-it-py[linkify,plugins]->gradio) (0.3.3)\n",
+      "Requirement already satisfied: linkify-it-py~=1.0 in ./.local/lib/python3.8/site-packages (from markdown-it-py[linkify,plugins]->gradio) (1.0.3)\n",
+      "Requirement already satisfied: packaging>=20.0 in ./.local/lib/python3.8/site-packages (from matplotlib->gradio) (22.0)\n",
       "Requirement already satisfied: cycler>=0.10 in /usr/lib/python3/dist-packages (from matplotlib->gradio) (0.10.0)\n",
-      "Requirement already satisfied: fonttools>=4.22.0 in ./.local/lib/python3.8/site-packages (from matplotlib->gradio) (4.38.0)\n",
       "Requirement already satisfied: pyparsing>=2.2.1 in /usr/lib/python3/dist-packages (from matplotlib->gradio) (2.4.6)\n",
+      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/lib/python3/dist-packages (from matplotlib->gradio) (1.0.1)\n",
+      "Requirement already satisfied: fonttools>=4.22.0 in ./.local/lib/python3.8/site-packages (from matplotlib->gradio) (4.38.0)\n",
       "Requirement already satisfied: python-dateutil>=2.7 in ./.local/lib/python3.8/site-packages (from matplotlib->gradio) (2.8.2)\n",
-      "Requirement already satisfied: packaging>=20.0 in ./.local/lib/python3.8/site-packages (from matplotlib->gradio) (21.3)\n",
       "Requirement already satisfied: pytz>=2020.1 in ./.local/lib/python3.8/site-packages (from pandas->gradio) (2022.5)\n",
-      "Requirement already satisfied: cryptography>=2.5 in /usr/lib/python3/dist-packages (from paramiko->gradio) (2.8)\n",
       "Requirement already satisfied: pynacl>=1.0.1 in /usr/lib/python3/dist-packages (from paramiko->gradio) (1.3.0)\n",
       "Requirement already satisfied: six in /usr/lib/python3/dist-packages (from paramiko->gradio) (1.14.0)\n",
-      "Requirement already satisfied: bcrypt>=3.1.3 in /usr/local/lib/python3.8/dist-packages (from paramiko->gradio) (4.0.1)\n",
+      "Requirement already satisfied: cryptography>=2.5 in /usr/lib/python3/dist-packages (from paramiko->gradio) (2.8)\n",
+      "Requirement already satisfied: bcrypt>=3.1.3 in ./.local/lib/python3.8/site-packages (from paramiko->gradio) (4.0.1)\n",
       "Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.local/lib/python3.8/site-packages (from requests->gradio) (1.26.13)\n",
       "Requirement already satisfied: idna<4,>=2.5 in ./.local/lib/python3.8/site-packages (from requests->gradio) (3.4)\n",
       "Requirement already satisfied: click>=7.0 in /usr/lib/python3/dist-packages (from uvicorn->gradio) (7.0)\n",
-      "Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.8/dist-packages (from linkify-it-py~=1.0->markdown-it-py[linkify,plugins]->gradio) (1.0.1)\n",
-      "Defaulting to user installation because normal site-packages is not writeable\n",
-      "Requirement already satisfied: more-itertools in /usr/lib/python3/dist-packages (4.2.0)\n"
+      "Requirement already satisfied: uc-micro-py in ./.local/lib/python3.8/site-packages (from linkify-it-py~=1.0->markdown-it-py[linkify,plugins]->gradio) (1.0.1)\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mDefaulting to user installation because normal site-packages is not writeable\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0mRequirement already satisfied: more-itertools in /usr/local/lib/python3.8/dist-packages (9.0.0)\n",
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m"
      ]
     }
    ],
    "source": [
     "!pip install git+https://github.com/huggingface/datasets\n",
     "!pip install git+https://github.com/huggingface/transformers\n",
-    "!pip3 install numexpr>=2.7.1\n",
+    "!pip3 install numexpr>=2.7.3\n",
     "!pip install librosa\n",
     "!pip install evaluate>=0.3.0\n",
     "!pip install jiwer\n",
@@ -636,7 +633,7 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "cee65d4b203d4b2a910d65aba8ff273c",
+       "model_id": "eb5276b0c5844a94939f868f9c37e908",
        "version_major": 2,
        "version_minor": 0
       },
@@ -652,7 +649,8 @@
     "from huggingface_hub import notebook_login\n",
     "\n",
     "notebook_login()\n",
-    "\n"
+    "\n",
+    "# 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"
    ]
   },
   {
@@ -725,15 +723,22 @@
    },
    "outputs": [
     {
-     "name": "stderr",
+     "name": "stdout",
      "output_type": "stream",
      "text": [
-      "/home/ubuntu/.local/lib/python3.8/site-packages/pandas/core/computation/expressions.py:20: UserWarning: Pandas requires version '2.7.3' or newer of 'numexpr' (version '2.7.1' currently installed).\n",
-      "  from pandas.core.computation.check import NUMEXPR_INSTALLED\n"
+      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/home/ubuntu/.local/lib/python3.8/site-packages)\u001b[0m\u001b[33m\n",
+      "\u001b[0m"
      ]
     }
    ],
    "source": [
+    "!pip3 install --upgrade numexpr>=2.7.3\n",
+    "\n",
     "from datasets import Audio, interleave_datasets, IterableDataset, load_dataset\n",
     "from typing import List, Optional\n",
     "\n",
@@ -895,7 +900,8 @@
    "execution_count": 11,
    "id": "77d9f0c5-8607-4642-a8ac-c3ab2e223ea6",
    "metadata": {
-    "id": "77d9f0c5-8607-4642-a8ac-c3ab2e223ea6"
+    "id": "77d9f0c5-8607-4642-a8ac-c3ab2e223ea6",
+    "tags": []
    },
    "outputs": [],
    "source": [
@@ -1448,6 +1454,7 @@
    "source": [
     "from transformers import WhisperForConditionalGeneration\n",
     "\n",
+    "#model = WhisperForConditionalGeneration.from_pretrained(\"emilios/whisper-medium-el-wer-14\")\n",
     "model = WhisperForConditionalGeneration.from_pretrained(\"emilios/whisper-medium-el\")\n",
     "#model = WhisperForConditionalGeneration.from_pretrained(\"farsipal/whisper-small-el\")\n",
     "\n",
@@ -1500,7 +1507,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 45,
+   "execution_count": 26,
    "id": "0ae3e9af-97b7-4aa0-ae85-20b23b5bcb3a",
    "metadata": {
     "colab": {
@@ -1509,15 +1516,7 @@
     "id": "0ae3e9af-97b7-4aa0-ae85-20b23b5bcb3a",
     "outputId": "7290f729-fb46-4190-dcc5-5cf01e1c9808"
    },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "PyTorch: setting up devices\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from transformers import Seq2SeqTrainingArguments\n",
     "\n",
@@ -1525,14 +1524,14 @@
     "    output_dir=\"./whisper-medium-el\",  # your repo name\n",
     "    #output_dir=\"./whisper-small-el\",  # your repo name\n",
     "    per_device_train_batch_size=32,\n",
-    "    gradient_accumulation_steps=2,  # increase by 2x for every 2x decrease in batch size\n",
+    "    gradient_accumulation_steps=1,  # increase by 2x for every 2x decrease in batch size\n",
     "    learning_rate=1e-5,\n",
     "    warmup_steps=500,\n",
     "    max_steps=5000,\n",
     "    gradient_checkpointing=True,\n",
     "    fp16=True,\n",
     "    evaluation_strategy=\"steps\",\n",
-    "    per_device_eval_batch_size=8,\n",
+    "    per_device_eval_batch_size=16,\n",
     "    predict_with_generate=True,\n",
     "    generation_max_length=225,\n",
     "    save_steps=1000,\n",
@@ -1543,13 +1542,15 @@
     "    metric_for_best_model=\"wer\",\n",
     "    greater_is_better=False,\n",
     "    push_to_hub=True,\n",
+    "    #ignore_data_skip=True,\n",
+    "#    resume_from_checkpoint=\"checkpoint-4000\"\n",
     "    #optim=\"adamw_bnb_8bit\"\n",
     ")\n"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 46,
+   "execution_count": 27,
    "id": "o72eOpGzD_sK",
    "metadata": {
     "colab": {
@@ -1563,7 +1564,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Thu Dec  8 18:57:05 2022       \n",
+      "Mon Dec 12 01:25:00 2022       \n",
       "+-----------------------------------------------------------------------------+\n",
       "| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |\n",
       "|-------------------------------+----------------------+----------------------+\n",
@@ -1572,7 +1573,7 @@
       "|                               |                      |               MIG M. |\n",
       "|===============================+======================+======================|\n",
       "|   0  NVIDIA A100-SXM...  On   | 00000000:06:00.0 Off |                    0 |\n",
-      "| N/A   32C    P0    55W / 400W |  39261MiB / 40960MiB |      0%      Default |\n",
+      "| N/A   32C    P0    44W / 400W |      2MiB / 40960MiB |      0%      Default |\n",
       "|                               |                      |             Disabled |\n",
       "+-------------------------------+----------------------+----------------------+\n",
       "                                                                               \n",
@@ -1581,7 +1582,7 @@
       "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
       "|        ID   ID                                                   Usage      |\n",
       "|=============================================================================|\n",
-      "|    0   N/A  N/A    130327      C   /usr/bin/python3                39259MiB |\n",
+      "|  No running processes found                                                 |\n",
       "+-----------------------------------------------------------------------------+\n"
      ]
     }
@@ -1613,7 +1614,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 47,
+   "execution_count": 28,
    "id": "3ac16b62-b3c0-4c68-8f3d-9ecf471534b2",
    "metadata": {
     "id": "3ac16b62-b3c0-4c68-8f3d-9ecf471534b2"
@@ -1646,7 +1647,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 48,
+   "execution_count": 29,
    "id": "d546d7fe-0543-479a-b708-2ebabec19493",
    "metadata": {
     "colab": {
@@ -2217,8 +2218,29 @@
      ]
     },
     "id": "d546d7fe-0543-479a-b708-2ebabec19493",
-    "outputId": "2c83cd5c-b1cc-4a3b-8d6c-57239408a297"
+    "outputId": "2c83cd5c-b1cc-4a3b-8d6c-57239408a297",
+    "tags": []
    },
+   "outputs": [],
+   "source": [
+    "#from huggingface_hub import Repository\n",
+    "#repo = Repository(local_dir=\"huggingface-hub\", clone_from=\"emilios/whisper-medium-el\")\n",
+    "#repo.git_pull()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bfa35273-00f8-49c1-846a-d413cd2f072d",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "id": "7118041a-5ff8-48ae-b02b-3efdb0d13490",
+   "metadata": {},
    "outputs": [
     {
      "name": "stderr",
@@ -2231,6 +2253,7 @@
     }
    ],
    "source": [
+    "\n",
     "from transformers import Seq2SeqTrainer\n",
     "\n",
     "#import torch._dynamo as dynamo\n",
@@ -2260,7 +2283,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 49,
+   "execution_count": 31,
    "id": "a1ccb9ed-cbc8-4419-91c0-651e9424b672",
    "metadata": {
     "id": "a1ccb9ed-cbc8-4419-91c0-651e9424b672"
@@ -2343,7 +2366,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 32,
    "id": "ee8b7b8e-1c9a-4d77-9137-1778a629e6de",
    "metadata": {
     "id": "ee8b7b8e-1c9a-4d77-9137-1778a629e6de"
@@ -2353,22 +2376,118 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
+      "Loading model from ./whisper-medium-el/checkpoint-4000.\n",
       "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
       "  warnings.warn(\n",
       "***** Running training *****\n",
-      "  Num examples = 320000\n",
+      "  Num examples = 160000\n",
       "  Num Epochs = 9223372036854775807\n",
       "  Instantaneous batch size per device = 32\n",
-      "  Total train batch size (w. parallel, distributed & accumulation) = 64\n",
-      "  Gradient Accumulation steps = 2\n",
+      "  Total train batch size (w. parallel, distributed & accumulation) = 32\n",
+      "  Gradient Accumulation steps = 1\n",
       "  Total optimization steps = 5000\n",
       "  Number of trainable parameters = 763857920\n",
-      "Reading metadata...: 1914it [00:00, 13829.19it/s]\n"
+      "  Continuing training from checkpoint, will skip to saved global_step\n",
+      "  Continuing training from epoch 0\n",
+      "  Continuing training from global step 4000\n",
+      "Reading metadata...: 1914it [00:00, 10645.75it/s]\n",
+      "The following columns in the training set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "\n",
+       "    <div>\n",
+       "      \n",
+       "      <progress value='5001' max='5000' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
+       "      [5000/5000 1:46:44, Epoch 4.04/9223372036854775807]\n",
+       "    </div>\n",
+       "    <table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       " <tr style=\"text-align: left;\">\n",
+       "      <th>Step</th>\n",
+       "      <th>Training Loss</th>\n",
+       "      <th>Validation Loss</th>\n",
+       "      <th>Wer</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <td>5000</td>\n",
+       "      <td>0.002400</td>\n",
+       "      <td>0.392416</td>\n",
+       "      <td>12.444279</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table><p>"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Reading metadata...: 1914it [00:00, 49446.87it/s]\n",
+      "Reading metadata...: 1914it [00:01, 1168.90it/s]\n",
+      "Reading metadata...: 1914it [00:00, 53478.68it/s]\n",
+      "Reading metadata...: 1914it [00:00, 56742.28it/s]\n",
+      "Reading metadata...: 1914it [00:00, 43779.78it/s]\n",
+      "Reading metadata...: 1914it [00:00, 47903.73it/s]\n",
+      "Reading metadata...: 1914it [00:00, 41580.46it/s]\n",
+      "Reading metadata...: 1914it [00:00, 45482.55it/s]\n",
+      "Reading metadata...: 1914it [00:00, 35803.98it/s]\n",
+      "***** Running Evaluation *****\n",
+      "  Num examples: Unknown\n",
+      "  Batch size = 16\n",
+      "Reading metadata...: 1696it [00:00, 12322.32it/s]\n",
+      "The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: down_votes, up_votes, segment, gender, accent, locale, input_length, path, age, client_id. If down_votes, up_votes, segment, gender, accent, locale, input_length, path, age, client_id are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.\n",
+      "Saving model checkpoint to ./whisper-medium-el/checkpoint-5000\n",
+      "Configuration saved in ./whisper-medium-el/checkpoint-5000/config.json\n",
+      "Model weights saved in ./whisper-medium-el/checkpoint-5000/pytorch_model.bin\n",
+      "Feature extractor saved in ./whisper-medium-el/checkpoint-5000/preprocessor_config.json\n",
+      "tokenizer config file saved in ./whisper-medium-el/checkpoint-5000/tokenizer_config.json\n",
+      "Special tokens file saved in ./whisper-medium-el/checkpoint-5000/special_tokens_map.json\n",
+      "added tokens file saved in ./whisper-medium-el/checkpoint-5000/added_tokens.json\n",
+      "Feature extractor saved in ./whisper-medium-el/preprocessor_config.json\n",
+      "tokenizer config file saved in ./whisper-medium-el/tokenizer_config.json\n",
+      "Special tokens file saved in ./whisper-medium-el/special_tokens_map.json\n",
+      "added tokens file saved in ./whisper-medium-el/added_tokens.json\n"
+     ]
+    },
+    {
+     "ename": "OSError",
+     "evalue": "error: insufficient permission for adding an object to repository database .git/objects\nerror: runs/Dec12_01-10-01_150-136-33-0/events.out.tfevents.1670807412.150-136-33-0.2756654.4: failed to insert into database\nerror: unable to index file 'runs/Dec12_01-10-01_150-136-33-0/events.out.tfevents.1670807412.150-136-33-0.2756654.4'\nfatal: adding files failed\nEncountered 1 file(s) that may not have been copied correctly on Windows:\n\tcheckpoint-5000/optimizer.pt\n\nSee: `git lfs help smudge` for more details.\n",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mCalledProcessError\u001b[0m                        Traceback (most recent call last)",
+      "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/huggingface_hub/repository.py\u001b[0m in \u001b[0;36mgit_add\u001b[0;34m(self, pattern, auto_lfs_track)\u001b[0m\n\u001b[1;32m   1121\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1122\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrun_subprocess\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"git add -v\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mpattern\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlocal_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1123\u001b[0m             \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Adding to index:\\n{result.stdout}\\n\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_subprocess.py\u001b[0m in \u001b[0;36mrun_subprocess\u001b[0;34m(command, folder, check, **kwargs)\u001b[0m\n\u001b[1;32m     60\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 61\u001b[0;31m     return subprocess.run(\n\u001b[0m\u001b[1;32m     62\u001b[0m         \u001b[0mcommand\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m/usr/lib/python3.8/subprocess.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(input, capture_output, timeout, check, *popenargs, **kwargs)\u001b[0m\n\u001b[1;32m    515\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcheck\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mretcode\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 516\u001b[0;31m             raise CalledProcessError(retcode, process.args,\n\u001b[0m\u001b[1;32m    517\u001b[0m                                      output=stdout, stderr=stderr)\n",
+      "\u001b[0;31mCalledProcessError\u001b[0m: Command '['git', 'add', '-v', '.']' returned non-zero exit status 128.",
+      "\nDuring handling of the above exception, another exception occurred:\n",
+      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
+      "\u001b[0;32m<ipython-input-32-d2e90939d965>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#trainer.train()\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtrainer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresume_from_checkpoint\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
+      "\u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/trainer.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)\u001b[0m\n\u001b[1;32m   1533\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_inner_training_loop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_train_batch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mauto_find_batch_size\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1534\u001b[0m         )\n\u001b[0;32m-> 1535\u001b[0;31m         return inner_training_loop(\n\u001b[0m\u001b[1;32m   1536\u001b[0m             \u001b[0margs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1537\u001b[0m             \u001b[0mresume_from_checkpoint\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mresume_from_checkpoint\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/trainer.py\u001b[0m in \u001b[0;36m_inner_training_loop\u001b[0;34m(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)\u001b[0m\n\u001b[1;32m   1858\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontrol\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallback_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_step_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontrol\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1859\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1860\u001b[0;31m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_log_save_evaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtr_loss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepoch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mignore_keys_for_eval\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1861\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1862\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontrol\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallback_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_substep_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontrol\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/trainer.py\u001b[0m in \u001b[0;36m_maybe_log_save_evaluate\u001b[0;34m(self, tr_loss, model, trial, epoch, ignore_keys_for_eval)\u001b[0m\n\u001b[1;32m   2125\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2126\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontrol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_save\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2127\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_save_checkpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrial\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetrics\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmetrics\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2128\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontrol\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallback_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_save\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontrol\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2129\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/trainer.py\u001b[0m in \u001b[0;36m_save_checkpoint\u001b[0;34m(self, model, trial, metrics)\u001b[0m\n\u001b[1;32m   2269\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2270\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpush_to_hub\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2271\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_push_from_checkpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput_dir\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2272\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2273\u001b[0m         \u001b[0;31m# Maybe delete some older checkpoints.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;32m~/.local/lib/python3.8/site-packages/transformers/trainer.py\u001b[0m in \u001b[0;36m_push_from_checkpoint\u001b[0;34m(self, checkpoint_folder)\u001b[0m\n\u001b[1;32m   3441\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3442\u001b[0m                 \u001b[0mcommit_message\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mf\"Training in progress, epoch {int(self.state.epoch)}\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3443\u001b[0;31m             _, self.push_in_progress = self.repo.push_to_hub(\n\u001b[0m\u001b[1;32m   3444\u001b[0m                 \u001b[0mcommit_message\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcommit_message\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mblocking\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mauto_lfs_prune\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3445\u001b[0m             )\n",
+      "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/huggingface_hub/repository.py\u001b[0m in \u001b[0;36mpush_to_hub\u001b[0;34m(self, commit_message, blocking, clean_ok, auto_lfs_prune)\u001b[0m\n\u001b[1;32m   1428\u001b[0m             \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Repo currently clean. Ignoring push_to_hub\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1429\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1430\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgit_add\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mauto_lfs_track\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1431\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgit_commit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommit_message\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1432\u001b[0m         return self.git_push(\n",
+      "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/huggingface_hub/repository.py\u001b[0m in \u001b[0;36mgit_add\u001b[0;34m(self, pattern, auto_lfs_track)\u001b[0m\n\u001b[1;32m   1123\u001b[0m             \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Adding to index:\\n{result.stdout}\\n\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1124\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0msubprocess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mCalledProcessError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1125\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mEnvironmentError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstderr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1126\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1127\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mgit_commit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcommit_message\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"commit files to HF hub\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;31mOSError\u001b[0m: error: insufficient permission for adding an object to repository database .git/objects\nerror: runs/Dec12_01-10-01_150-136-33-0/events.out.tfevents.1670807412.150-136-33-0.2756654.4: failed to insert into database\nerror: unable to index file 'runs/Dec12_01-10-01_150-136-33-0/events.out.tfevents.1670807412.150-136-33-0.2756654.4'\nfatal: adding files failed\nEncountered 1 file(s) that may not have been copied correctly on Windows:\n\tcheckpoint-5000/optimizer.pt\n\nSee: `git lfs help smudge` for more details.\n"
      ]
     }
    ],
    "source": [
-    "trainer.train()"
+    "#trainer.train()\n",
+    "trainer.train(resume_from_checkpoint = True)\n"
    ]
   },
   {
@@ -2396,7 +2515,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 33,
    "id": "6dd0e310-9b07-4133-ac14-2ed2d7524e22",
    "metadata": {
     "id": "6dd0e310-9b07-4133-ac14-2ed2d7524e22"
@@ -2409,7 +2528,7 @@
     "    \"dataset\": \"Common Voice 11.0\",  # a 'pretty' name for the training dataset\n",
     "    #\"dataset\": \"Google FLEURS\",  # a 'pretty' name for the training dataset\n",
     "    \"language\": \"el\",\n",
-    "    \"model_name\": \"Whisper Medium El - Greek One\",  # a 'pretty' name for your model\n",
+    "    \"model_name\": \"Whisper Medium El Greco Greek\",  # a 'pretty' name for your model\n",
     "    \"finetuned_from\": \"openai/whisper-medium\",\n",
     "    \"tasks\": \"automatic-speech-recognition\",\n",
     "    \"tags\": \"hf-asr-leaderboard, whisper-medium, mozilla-foundation/common_voice_11_0, greek, whisper-event\",\n",
@@ -2428,15 +2547,50 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 35,
    "id": "95737cda-c5dd-4887-a4d0-dfcb0d61d977",
    "metadata": {
     "id": "95737cda-c5dd-4887-a4d0-dfcb0d61d977"
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Saving model checkpoint to ./whisper-medium-el\n",
+      "Configuration saved in ./whisper-medium-el/config.json\n",
+      "Model weights saved in ./whisper-medium-el/pytorch_model.bin\n",
+      "Feature extractor saved in ./whisper-medium-el/preprocessor_config.json\n",
+      "tokenizer config file saved in ./whisper-medium-el/tokenizer_config.json\n",
+      "Special tokens file saved in ./whisper-medium-el/special_tokens_map.json\n",
+      "added tokens file saved in ./whisper-medium-el/added_tokens.json\n",
+      "Dropping the following result as it does not have all the necessary fields:\n",
+      "{'task': {'name': 'Automatic Speech Recognition', 'type': 'automatic-speech-recognition'}, 'dataset': {'name': 'Common Voice 11.0', 'type': 'mozilla-foundation/common_voice_11_0', 'config': 'el', 'split': 'test', 'args': 'el'}}\n",
+      "To https://huggingface.co/emilios/whisper-medium-el\n",
+      "   761a953..1303559  main -> main\n",
+      "\n"
+     ]
+    }
+   ],
    "source": [
     "trainer.push_to_hub(**kwargs)"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e7a702f1-f48f-4d2a-8139-89fd67308d94",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b8327ca7-cb48-4fd3-90d6-bb5097f8f307",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {

eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 12.44,
+    "eval_loss": 0.38012266159057617,
+    "eval_runtime": 1110.8612,
+    "eval_samples_per_second": 1.527,
+    "eval_steps_per_second": 0.095,
+    "eval_wer": 11.199851411589897
+}

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2ea24fb6272350e35c7f819c3836f437778f46ef44aef6816a93d4a3869b9cef
+oid sha256:aa9c1cc306eaa444f99ef35b5e3ffb717f457ea531fd3a8d4727ffdbd4b4cccd
 size 3055754841

run_inter_1gpu.sh

@@ -1,10 +1,16 @@
-python run_interleave.py --model_name_or_path="emilios/whisper-medium-el" \
+python run_interleave.py \
+--model_name_or_path="emilios/whisper-medium-el" \
+--language="greek" \
 --dataset_name="mozilla-foundation/common_voice_11_0,google/fleurs" \
 --dataset_config_name="el,el_gr" \
---language="greek" \
---train_split_name="train+validation" \
---eval_split_name="test" \
---model_index_name="Whisper Medium El - Greek One" \
+--train_split_name="train+validation,train+validation" \
+--eval_split_name="test,-" \
+--text_column_name  'sentence,transcription' \
+--audio_column_name 'audio,audio' \
+--task="transcribe" \
+--streaming="False" \
+--model_index_name="Whisper Medium El Greco" \
+--dropout="0.1" \
 --max_steps="5000" \
 --output_dir="./" \
 --per_device_train_batch_size="32" \
@@ -33,6 +39,5 @@ python run_interleave.py --model_name_or_path="emilios/whisper-medium-el" \
 --do_eval \
 --predict_with_generate \
 --do_normalize_eval \
---streaming \
 --use_auth_token \
 --push_to_hub

run_interleave.py

@@ -60,6 +60,7 @@ from transformers.utils.versions import require_version
 from transformers.models.whisper.english_normalizer import BasicTextNormalizer
 
 TEXT_COL_NAME="text"
+TEXT_COL_NAME="sentence,transcription"
 AUDIO_COL_NAME="audio"
 
 # Will error if the minimal version of Transformers is not installed. Remove at your own risks.

run_whisper-md-el-intlv-xs.sh

@@ -1,13 +1,11 @@
-# whisper-md-el-intlv-xs
-python run_speech_recognition_seq2seq_streaming.py \
-                --model_name_or_path   'openai/whisper-medium' \
+python run_interleave.py \
+                --model_name_or_path   'emilios/whisper-medium-el' \
                 --model_revision   main \
                 --do_train   True \
                 --do_eval   True \
-                --use_auth_token   False \
                 --freeze_feature_encoder   False \
                 --freeze_encoder   False \
-                --model_index_name   'whisper-md-el-intlv-xs' \
+                --model_index_name   'Whisper Medium El Greco' \
                 --dataset_name 'mozilla-foundation/common_voice_11_0,google/fleurs' \
                 --dataset_config_name 'el,el_gr' \
                 --train_split_name  'train+validation,train+validation' \
@@ -22,10 +20,11 @@ python run_speech_recognition_seq2seq_streaming.py \
                 --language   greek \
                 --task transcribe \
                 --shuffle_buffer_size   500 \
-                --output_dir   './data/finetuningRuns/whisper-md-el-intlv-xs' \
+                --output_dir   './' \
                 --overwrite_output_dir   True \
                 --per_device_train_batch_size   32 \
-                --gradient_accumulation_steps  2 \
+                --gradient_accumulation_steps  1 \
+                --per_device_eval_batch_size   16 \
                 --learning_rate   1e-5 \
                 --dropout         0.1 \
                 --warmup_steps   500 \
@@ -35,7 +34,6 @@ python run_speech_recognition_seq2seq_streaming.py \
                 --cache_dir   '~/.cache' \
                 --fp16   True \
                 --evaluation_strategy   steps \
-                --per_device_eval_batch_size   8 \
                 --predict_with_generate   True \
                 --generation_max_length   225 \
                 --save_steps   1000 \

runs/Dec12_03-57-18_150-136-33-0/1670819499.156932/events.out.tfevents.1670819499.150-136-33-0.3405254.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08d8d9887d71b5ab49ebfe74af431c20dbdf9f4008efef745317368d7cd9a003
+size 5862

runs/Dec12_03-57-18_150-136-33-0/events.out.tfevents.1670819499.150-136-33-0.3405254.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:982a32db879d80cb27cd4cd9a1268b9a39ff843b045a1882012739dc4d6da91a
+size 37595

runs/Dec12_03-57-18_150-136-33-0/events.out.tfevents.1670847754.150-136-33-0.3405254.2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4e8c731d53f21751b9ee9166ca526802d8a90c459680e2c59a3e71b3f256bc0
+size 358

train_results.json

@@ -0,0 +1,7 @@
+{
+    "epoch": 12.44,
+    "train_loss": 0.020854657278954983,
+    "train_runtime": 27142.1177,
+    "train_samples_per_second": 5.895,
+    "train_steps_per_second": 0.184
+}

trainer_state.json

@@ -0,0 +1,1270 @@
+{
+  "best_metric": 11.199851411589897,
+  "best_model_checkpoint": "./checkpoint-5000",
+  "epoch": 12.437810945273633,
+  "global_step": 5000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.06,
+      "learning_rate": 4.800000000000001e-07,
+      "loss": 1.0869,
+      "step": 25
+    },
+    {
+      "epoch": 0.12,
+      "learning_rate": 9.800000000000001e-07,
+      "loss": 0.588,
+      "step": 50
+    },
+    {
+      "epoch": 0.19,
+      "learning_rate": 1.48e-06,
+      "loss": 0.2266,
+      "step": 75
+    },
+    {
+      "epoch": 0.25,
+      "learning_rate": 1.98e-06,
+      "loss": 0.1899,
+      "step": 100
+    },
+    {
+      "epoch": 0.31,
+      "learning_rate": 2.4800000000000004e-06,
+      "loss": 0.1476,
+      "step": 125
+    },
+    {
+      "epoch": 0.37,
+      "learning_rate": 2.9800000000000003e-06,
+      "loss": 0.111,
+      "step": 150
+    },
+    {
+      "epoch": 0.44,
+      "learning_rate": 3.48e-06,
+      "loss": 0.0958,
+      "step": 175
+    },
+    {
+      "epoch": 0.5,
+      "learning_rate": 3.980000000000001e-06,
+      "loss": 0.0904,
+      "step": 200
+    },
+    {
+      "epoch": 0.56,
+      "learning_rate": 4.48e-06,
+      "loss": 0.0783,
+      "step": 225
+    },
+    {
+      "epoch": 0.62,
+      "learning_rate": 4.980000000000001e-06,
+      "loss": 0.0676,
+      "step": 250
+    },
+    {
+      "epoch": 0.68,
+      "learning_rate": 5.480000000000001e-06,
+      "loss": 0.0612,
+      "step": 275
+    },
+    {
+      "epoch": 0.75,
+      "learning_rate": 5.98e-06,
+      "loss": 0.0503,
+      "step": 300
+    },
+    {
+      "epoch": 0.81,
+      "learning_rate": 6.480000000000001e-06,
+      "loss": 0.0523,
+      "step": 325
+    },
+    {
+      "epoch": 0.87,
+      "learning_rate": 6.98e-06,
+      "loss": 0.045,
+      "step": 350
+    },
+    {
+      "epoch": 0.93,
+      "learning_rate": 7.48e-06,
+      "loss": 0.0395,
+      "step": 375
+    },
+    {
+      "epoch": 1.0,
+      "learning_rate": 7.980000000000002e-06,
+      "loss": 0.0454,
+      "step": 400
+    },
+    {
+      "epoch": 1.06,
+      "learning_rate": 8.48e-06,
+      "loss": 0.0336,
+      "step": 425
+    },
+    {
+      "epoch": 1.12,
+      "learning_rate": 8.98e-06,
+      "loss": 0.0325,
+      "step": 450
+    },
+    {
+      "epoch": 1.18,
+      "learning_rate": 9.48e-06,
+      "loss": 0.0315,
+      "step": 475
+    },
+    {
+      "epoch": 1.24,
+      "learning_rate": 9.980000000000001e-06,
+      "loss": 0.0326,
+      "step": 500
+    },
+    {
+      "epoch": 1.31,
+      "learning_rate": 9.946666666666667e-06,
+      "loss": 0.0326,
+      "step": 525
+    },
+    {
+      "epoch": 1.37,
+      "learning_rate": 9.891111111111113e-06,
+      "loss": 0.0278,
+      "step": 550
+    },
+    {
+      "epoch": 1.43,
+      "learning_rate": 9.835555555555556e-06,
+      "loss": 0.0292,
+      "step": 575
+    },
+    {
+      "epoch": 1.49,
+      "learning_rate": 9.780000000000001e-06,
+      "loss": 0.0284,
+      "step": 600
+    },
+    {
+      "epoch": 1.55,
+      "learning_rate": 9.724444444444445e-06,
+      "loss": 0.0336,
+      "step": 625
+    },
+    {
+      "epoch": 1.62,
+      "learning_rate": 9.66888888888889e-06,
+      "loss": 0.0291,
+      "step": 650
+    },
+    {
+      "epoch": 1.68,
+      "learning_rate": 9.613333333333335e-06,
+      "loss": 0.0274,
+      "step": 675
+    },
+    {
+      "epoch": 1.74,
+      "learning_rate": 9.557777777777777e-06,
+      "loss": 0.024,
+      "step": 700
+    },
+    {
+      "epoch": 1.8,
+      "learning_rate": 9.502222222222223e-06,
+      "loss": 0.0254,
+      "step": 725
+    },
+    {
+      "epoch": 1.87,
+      "learning_rate": 9.446666666666667e-06,
+      "loss": 0.0201,
+      "step": 750
+    },
+    {
+      "epoch": 1.93,
+      "learning_rate": 9.391111111111111e-06,
+      "loss": 0.0261,
+      "step": 775
+    },
+    {
+      "epoch": 1.99,
+      "learning_rate": 9.335555555555557e-06,
+      "loss": 0.0235,
+      "step": 800
+    },
+    {
+      "epoch": 2.05,
+      "learning_rate": 9.280000000000001e-06,
+      "loss": 0.0165,
+      "step": 825
+    },
+    {
+      "epoch": 2.11,
+      "learning_rate": 9.224444444444445e-06,
+      "loss": 0.0148,
+      "step": 850
+    },
+    {
+      "epoch": 2.18,
+      "learning_rate": 9.168888888888889e-06,
+      "loss": 0.0161,
+      "step": 875
+    },
+    {
+      "epoch": 2.24,
+      "learning_rate": 9.113333333333335e-06,
+      "loss": 0.0144,
+      "step": 900
+    },
+    {
+      "epoch": 2.3,
+      "learning_rate": 9.057777777777779e-06,
+      "loss": 0.016,
+      "step": 925
+    },
+    {
+      "epoch": 2.36,
+      "learning_rate": 9.002222222222223e-06,
+      "loss": 0.0151,
+      "step": 950
+    },
+    {
+      "epoch": 2.43,
+      "learning_rate": 8.946666666666669e-06,
+      "loss": 0.0136,
+      "step": 975
+    },
+    {
+      "epoch": 2.49,
+      "learning_rate": 8.891111111111111e-06,
+      "loss": 0.0176,
+      "step": 1000
+    },
+    {
+      "epoch": 2.49,
+      "eval_loss": 0.2945367693901062,
+      "eval_runtime": 1114.0569,
+      "eval_samples_per_second": 1.522,
+      "eval_steps_per_second": 0.095,
+      "eval_wer": 12.611441307578009,
+      "step": 1000
+    },
+    {
+      "epoch": 2.55,
+      "learning_rate": 8.835555555555557e-06,
+      "loss": 0.0127,
+      "step": 1025
+    },
+    {
+      "epoch": 2.61,
+      "learning_rate": 8.78e-06,
+      "loss": 0.015,
+      "step": 1050
+    },
+    {
+      "epoch": 2.67,
+      "learning_rate": 8.724444444444445e-06,
+      "loss": 0.0134,
+      "step": 1075
+    },
+    {
+      "epoch": 2.74,
+      "learning_rate": 8.66888888888889e-06,
+      "loss": 0.0128,
+      "step": 1100
+    },
+    {
+      "epoch": 2.8,
+      "learning_rate": 8.613333333333333e-06,
+      "loss": 0.0132,
+      "step": 1125
+    },
+    {
+      "epoch": 2.86,
+      "learning_rate": 8.557777777777778e-06,
+      "loss": 0.0121,
+      "step": 1150
+    },
+    {
+      "epoch": 2.92,
+      "learning_rate": 8.502222222222223e-06,
+      "loss": 0.0121,
+      "step": 1175
+    },
+    {
+      "epoch": 2.99,
+      "learning_rate": 8.446666666666668e-06,
+      "loss": 0.012,
+      "step": 1200
+    },
+    {
+      "epoch": 3.05,
+      "learning_rate": 8.391111111111112e-06,
+      "loss": 0.0098,
+      "step": 1225
+    },
+    {
+      "epoch": 3.11,
+      "learning_rate": 8.335555555555556e-06,
+      "loss": 0.0092,
+      "step": 1250
+    },
+    {
+      "epoch": 3.17,
+      "learning_rate": 8.28e-06,
+      "loss": 0.0108,
+      "step": 1275
+    },
+    {
+      "epoch": 3.23,
+      "learning_rate": 8.224444444444444e-06,
+      "loss": 0.0083,
+      "step": 1300
+    },
+    {
+      "epoch": 3.3,
+      "learning_rate": 8.16888888888889e-06,
+      "loss": 0.0084,
+      "step": 1325
+    },
+    {
+      "epoch": 3.36,
+      "learning_rate": 8.113333333333334e-06,
+      "loss": 0.0087,
+      "step": 1350
+    },
+    {
+      "epoch": 3.42,
+      "learning_rate": 8.057777777777778e-06,
+      "loss": 0.0086,
+      "step": 1375
+    },
+    {
+      "epoch": 3.48,
+      "learning_rate": 8.002222222222222e-06,
+      "loss": 0.0087,
+      "step": 1400
+    },
+    {
+      "epoch": 3.54,
+      "learning_rate": 7.946666666666666e-06,
+      "loss": 0.009,
+      "step": 1425
+    },
+    {
+      "epoch": 3.61,
+      "learning_rate": 7.891111111111112e-06,
+      "loss": 0.0099,
+      "step": 1450
+    },
+    {
+      "epoch": 3.67,
+      "learning_rate": 7.835555555555556e-06,
+      "loss": 0.0079,
+      "step": 1475
+    },
+    {
+      "epoch": 3.73,
+      "learning_rate": 7.78e-06,
+      "loss": 0.0083,
+      "step": 1500
+    },
+    {
+      "epoch": 3.79,
+      "learning_rate": 7.724444444444446e-06,
+      "loss": 0.0067,
+      "step": 1525
+    },
+    {
+      "epoch": 3.86,
+      "learning_rate": 7.66888888888889e-06,
+      "loss": 0.0083,
+      "step": 1550
+    },
+    {
+      "epoch": 3.92,
+      "learning_rate": 7.613333333333334e-06,
+      "loss": 0.0095,
+      "step": 1575
+    },
+    {
+      "epoch": 3.98,
+      "learning_rate": 7.557777777777779e-06,
+      "loss": 0.008,
+      "step": 1600
+    },
+    {
+      "epoch": 4.04,
+      "learning_rate": 7.502222222222223e-06,
+      "loss": 0.0066,
+      "step": 1625
+    },
+    {
+      "epoch": 4.1,
+      "learning_rate": 7.446666666666668e-06,
+      "loss": 0.0065,
+      "step": 1650
+    },
+    {
+      "epoch": 4.17,
+      "learning_rate": 7.3911111111111125e-06,
+      "loss": 0.0063,
+      "step": 1675
+    },
+    {
+      "epoch": 4.23,
+      "learning_rate": 7.335555555555556e-06,
+      "loss": 0.0063,
+      "step": 1700
+    },
+    {
+      "epoch": 4.29,
+      "learning_rate": 7.280000000000001e-06,
+      "loss": 0.0057,
+      "step": 1725
+    },
+    {
+      "epoch": 4.35,
+      "learning_rate": 7.224444444444445e-06,
+      "loss": 0.0067,
+      "step": 1750
+    },
+    {
+      "epoch": 4.42,
+      "learning_rate": 7.1688888888888895e-06,
+      "loss": 0.0052,
+      "step": 1775
+    },
+    {
+      "epoch": 4.48,
+      "learning_rate": 7.113333333333334e-06,
+      "loss": 0.0069,
+      "step": 1800
+    },
+    {
+      "epoch": 4.54,
+      "learning_rate": 7.057777777777778e-06,
+      "loss": 0.0063,
+      "step": 1825
+    },
+    {
+      "epoch": 4.6,
+      "learning_rate": 7.0022222222222225e-06,
+      "loss": 0.0059,
+      "step": 1850
+    },
+    {
+      "epoch": 4.66,
+      "learning_rate": 6.946666666666667e-06,
+      "loss": 0.0053,
+      "step": 1875
+    },
+    {
+      "epoch": 4.73,
+      "learning_rate": 6.891111111111111e-06,
+      "loss": 0.0059,
+      "step": 1900
+    },
+    {
+      "epoch": 4.79,
+      "learning_rate": 6.835555555555556e-06,
+      "loss": 0.0063,
+      "step": 1925
+    },
+    {
+      "epoch": 4.85,
+      "learning_rate": 6.780000000000001e-06,
+      "loss": 0.0057,
+      "step": 1950
+    },
+    {
+      "epoch": 4.91,
+      "learning_rate": 6.724444444444444e-06,
+      "loss": 0.0056,
+      "step": 1975
+    },
+    {
+      "epoch": 4.98,
+      "learning_rate": 6.668888888888889e-06,
+      "loss": 0.0064,
+      "step": 2000
+    },
+    {
+      "epoch": 4.98,
+      "eval_loss": 0.34234580397605896,
+      "eval_runtime": 1111.8202,
+      "eval_samples_per_second": 1.525,
+      "eval_steps_per_second": 0.095,
+      "eval_wer": 12.230683506686479,
+      "step": 2000
+    },
+    {
+      "epoch": 5.04,
+      "learning_rate": 6.613333333333334e-06,
+      "loss": 0.0039,
+      "step": 2025
+    },
+    {
+      "epoch": 5.1,
+      "learning_rate": 6.557777777777778e-06,
+      "loss": 0.0036,
+      "step": 2050
+    },
+    {
+      "epoch": 5.16,
+      "learning_rate": 6.502222222222223e-06,
+      "loss": 0.0036,
+      "step": 2075
+    },
+    {
+      "epoch": 5.22,
+      "learning_rate": 6.446666666666668e-06,
+      "loss": 0.0039,
+      "step": 2100
+    },
+    {
+      "epoch": 5.29,
+      "learning_rate": 6.391111111111111e-06,
+      "loss": 0.004,
+      "step": 2125
+    },
+    {
+      "epoch": 5.35,
+      "learning_rate": 6.335555555555556e-06,
+      "loss": 0.0041,
+      "step": 2150
+    },
+    {
+      "epoch": 5.41,
+      "learning_rate": 6.280000000000001e-06,
+      "loss": 0.0046,
+      "step": 2175
+    },
+    {
+      "epoch": 5.47,
+      "learning_rate": 6.224444444444445e-06,
+      "loss": 0.0058,
+      "step": 2200
+    },
+    {
+      "epoch": 5.53,
+      "learning_rate": 6.16888888888889e-06,
+      "loss": 0.0043,
+      "step": 2225
+    },
+    {
+      "epoch": 5.6,
+      "learning_rate": 6.113333333333333e-06,
+      "loss": 0.0042,
+      "step": 2250
+    },
+    {
+      "epoch": 5.66,
+      "learning_rate": 6.057777777777778e-06,
+      "loss": 0.0045,
+      "step": 2275
+    },
+    {
+      "epoch": 5.72,
+      "learning_rate": 6.002222222222223e-06,
+      "loss": 0.0041,
+      "step": 2300
+    },
+    {
+      "epoch": 5.78,
+      "learning_rate": 5.946666666666668e-06,
+      "loss": 0.004,
+      "step": 2325
+    },
+    {
+      "epoch": 5.85,
+      "learning_rate": 5.891111111111112e-06,
+      "loss": 0.004,
+      "step": 2350
+    },
+    {
+      "epoch": 5.91,
+      "learning_rate": 5.8355555555555565e-06,
+      "loss": 0.0043,
+      "step": 2375
+    },
+    {
+      "epoch": 5.97,
+      "learning_rate": 5.78e-06,
+      "loss": 0.004,
+      "step": 2400
+    },
+    {
+      "epoch": 6.03,
+      "learning_rate": 5.724444444444445e-06,
+      "loss": 0.004,
+      "step": 2425
+    },
+    {
+      "epoch": 6.09,
+      "learning_rate": 5.6688888888888895e-06,
+      "loss": 0.0045,
+      "step": 2450
+    },
+    {
+      "epoch": 6.16,
+      "learning_rate": 5.613333333333334e-06,
+      "loss": 0.0049,
+      "step": 2475
+    },
+    {
+      "epoch": 6.22,
+      "learning_rate": 5.557777777777778e-06,
+      "loss": 0.0038,
+      "step": 2500
+    },
+    {
+      "epoch": 6.28,
+      "learning_rate": 5.5022222222222224e-06,
+      "loss": 0.0047,
+      "step": 2525
+    },
+    {
+      "epoch": 6.34,
+      "learning_rate": 5.4466666666666665e-06,
+      "loss": 0.0035,
+      "step": 2550
+    },
+    {
+      "epoch": 6.41,
+      "learning_rate": 5.391111111111111e-06,
+      "loss": 0.0032,
+      "step": 2575
+    },
+    {
+      "epoch": 6.47,
+      "learning_rate": 5.335555555555556e-06,
+      "loss": 0.0032,
+      "step": 2600
+    },
+    {
+      "epoch": 6.53,
+      "learning_rate": 5.28e-06,
+      "loss": 0.0032,
+      "step": 2625
+    },
+    {
+      "epoch": 6.59,
+      "learning_rate": 5.224444444444445e-06,
+      "loss": 0.0036,
+      "step": 2650
+    },
+    {
+      "epoch": 6.65,
+      "learning_rate": 5.168888888888889e-06,
+      "loss": 0.0035,
+      "step": 2675
+    },
+    {
+      "epoch": 6.72,
+      "learning_rate": 5.113333333333333e-06,
+      "loss": 0.0041,
+      "step": 2700
+    },
+    {
+      "epoch": 6.78,
+      "learning_rate": 5.057777777777778e-06,
+      "loss": 0.0035,
+      "step": 2725
+    },
+    {
+      "epoch": 6.84,
+      "learning_rate": 5.002222222222223e-06,
+      "loss": 0.0034,
+      "step": 2750
+    },
+    {
+      "epoch": 6.9,
+      "learning_rate": 4.946666666666667e-06,
+      "loss": 0.0039,
+      "step": 2775
+    },
+    {
+      "epoch": 6.97,
+      "learning_rate": 4.891111111111111e-06,
+      "loss": 0.0034,
+      "step": 2800
+    },
+    {
+      "epoch": 7.03,
+      "learning_rate": 4.835555555555556e-06,
+      "loss": 0.0035,
+      "step": 2825
+    },
+    {
+      "epoch": 7.09,
+      "learning_rate": 4.78e-06,
+      "loss": 0.0037,
+      "step": 2850
+    },
+    {
+      "epoch": 7.15,
+      "learning_rate": 4.724444444444445e-06,
+      "loss": 0.0026,
+      "step": 2875
+    },
+    {
+      "epoch": 7.21,
+      "learning_rate": 4.66888888888889e-06,
+      "loss": 0.0026,
+      "step": 2900
+    },
+    {
+      "epoch": 7.28,
+      "learning_rate": 4.613333333333334e-06,
+      "loss": 0.0034,
+      "step": 2925
+    },
+    {
+      "epoch": 7.34,
+      "learning_rate": 4.557777777777778e-06,
+      "loss": 0.0025,
+      "step": 2950
+    },
+    {
+      "epoch": 7.4,
+      "learning_rate": 4.502222222222223e-06,
+      "loss": 0.0023,
+      "step": 2975
+    },
+    {
+      "epoch": 7.46,
+      "learning_rate": 4.446666666666667e-06,
+      "loss": 0.0022,
+      "step": 3000
+    },
+    {
+      "epoch": 7.46,
+      "eval_loss": 0.3631705045700073,
+      "eval_runtime": 1107.3108,
+      "eval_samples_per_second": 1.532,
+      "eval_steps_per_second": 0.096,
+      "eval_wer": 11.589895988112927,
+      "step": 3000
+    },
+    {
+      "epoch": 7.52,
+      "learning_rate": 4.391111111111112e-06,
+      "loss": 0.0026,
+      "step": 3025
+    },
+    {
+      "epoch": 7.59,
+      "learning_rate": 4.3355555555555565e-06,
+      "loss": 0.0025,
+      "step": 3050
+    },
+    {
+      "epoch": 7.65,
+      "learning_rate": 4.2800000000000005e-06,
+      "loss": 0.0023,
+      "step": 3075
+    },
+    {
+      "epoch": 7.71,
+      "learning_rate": 4.2244444444444446e-06,
+      "loss": 0.0023,
+      "step": 3100
+    },
+    {
+      "epoch": 7.77,
+      "learning_rate": 4.168888888888889e-06,
+      "loss": 0.0023,
+      "step": 3125
+    },
+    {
+      "epoch": 7.84,
+      "learning_rate": 4.1133333333333335e-06,
+      "loss": 0.0027,
+      "step": 3150
+    },
+    {
+      "epoch": 7.9,
+      "learning_rate": 4.057777777777778e-06,
+      "loss": 0.0022,
+      "step": 3175
+    },
+    {
+      "epoch": 7.96,
+      "learning_rate": 4.002222222222222e-06,
+      "loss": 0.0025,
+      "step": 3200
+    },
+    {
+      "epoch": 8.02,
+      "learning_rate": 3.946666666666667e-06,
+      "loss": 0.0017,
+      "step": 3225
+    },
+    {
+      "epoch": 8.08,
+      "learning_rate": 3.891111111111111e-06,
+      "loss": 0.0016,
+      "step": 3250
+    },
+    {
+      "epoch": 8.15,
+      "learning_rate": 3.835555555555555e-06,
+      "loss": 0.0018,
+      "step": 3275
+    },
+    {
+      "epoch": 8.21,
+      "learning_rate": 3.7800000000000002e-06,
+      "loss": 0.0022,
+      "step": 3300
+    },
+    {
+      "epoch": 8.27,
+      "learning_rate": 3.724444444444445e-06,
+      "loss": 0.002,
+      "step": 3325
+    },
+    {
+      "epoch": 8.33,
+      "learning_rate": 3.668888888888889e-06,
+      "loss": 0.0018,
+      "step": 3350
+    },
+    {
+      "epoch": 8.4,
+      "learning_rate": 3.6133333333333336e-06,
+      "loss": 0.0015,
+      "step": 3375
+    },
+    {
+      "epoch": 8.46,
+      "learning_rate": 3.5577777777777785e-06,
+      "loss": 0.0022,
+      "step": 3400
+    },
+    {
+      "epoch": 8.52,
+      "learning_rate": 3.5022222222222225e-06,
+      "loss": 0.0022,
+      "step": 3425
+    },
+    {
+      "epoch": 8.58,
+      "learning_rate": 3.446666666666667e-06,
+      "loss": 0.0021,
+      "step": 3450
+    },
+    {
+      "epoch": 8.64,
+      "learning_rate": 3.391111111111111e-06,
+      "loss": 0.0022,
+      "step": 3475
+    },
+    {
+      "epoch": 8.71,
+      "learning_rate": 3.335555555555556e-06,
+      "loss": 0.0018,
+      "step": 3500
+    },
+    {
+      "epoch": 8.77,
+      "learning_rate": 3.2800000000000004e-06,
+      "loss": 0.002,
+      "step": 3525
+    },
+    {
+      "epoch": 8.83,
+      "learning_rate": 3.2244444444444444e-06,
+      "loss": 0.0025,
+      "step": 3550
+    },
+    {
+      "epoch": 8.89,
+      "learning_rate": 3.1688888888888893e-06,
+      "loss": 0.0017,
+      "step": 3575
+    },
+    {
+      "epoch": 8.96,
+      "learning_rate": 3.1133333333333337e-06,
+      "loss": 0.002,
+      "step": 3600
+    },
+    {
+      "epoch": 9.02,
+      "learning_rate": 3.0577777777777778e-06,
+      "loss": 0.0016,
+      "step": 3625
+    },
+    {
+      "epoch": 9.08,
+      "learning_rate": 3.0022222222222227e-06,
+      "loss": 0.0015,
+      "step": 3650
+    },
+    {
+      "epoch": 9.14,
+      "learning_rate": 2.946666666666667e-06,
+      "loss": 0.0019,
+      "step": 3675
+    },
+    {
+      "epoch": 9.2,
+      "learning_rate": 2.891111111111111e-06,
+      "loss": 0.0016,
+      "step": 3700
+    },
+    {
+      "epoch": 9.27,
+      "learning_rate": 2.835555555555556e-06,
+      "loss": 0.0016,
+      "step": 3725
+    },
+    {
+      "epoch": 9.33,
+      "learning_rate": 2.7800000000000005e-06,
+      "loss": 0.0018,
+      "step": 3750
+    },
+    {
+      "epoch": 9.39,
+      "learning_rate": 2.7244444444444445e-06,
+      "loss": 0.0014,
+      "step": 3775
+    },
+    {
+      "epoch": 9.45,
+      "learning_rate": 2.6688888888888894e-06,
+      "loss": 0.0013,
+      "step": 3800
+    },
+    {
+      "epoch": 9.51,
+      "learning_rate": 2.6133333333333334e-06,
+      "loss": 0.0015,
+      "step": 3825
+    },
+    {
+      "epoch": 9.58,
+      "learning_rate": 2.557777777777778e-06,
+      "loss": 0.0018,
+      "step": 3850
+    },
+    {
+      "epoch": 9.64,
+      "learning_rate": 2.5022222222222224e-06,
+      "loss": 0.0013,
+      "step": 3875
+    },
+    {
+      "epoch": 9.7,
+      "learning_rate": 2.446666666666667e-06,
+      "loss": 0.0015,
+      "step": 3900
+    },
+    {
+      "epoch": 9.76,
+      "learning_rate": 2.3911111111111113e-06,
+      "loss": 0.0013,
+      "step": 3925
+    },
+    {
+      "epoch": 9.83,
+      "learning_rate": 2.3355555555555557e-06,
+      "loss": 0.0014,
+      "step": 3950
+    },
+    {
+      "epoch": 9.89,
+      "learning_rate": 2.28e-06,
+      "loss": 0.0015,
+      "step": 3975
+    },
+    {
+      "epoch": 9.95,
+      "learning_rate": 2.2244444444444447e-06,
+      "loss": 0.0014,
+      "step": 4000
+    },
+    {
+      "epoch": 9.95,
+      "eval_loss": 0.37881383299827576,
+      "eval_runtime": 1112.8569,
+      "eval_samples_per_second": 1.524,
+      "eval_steps_per_second": 0.095,
+      "eval_wer": 11.255572065378901,
+      "step": 4000
+    },
+    {
+      "epoch": 10.01,
+      "learning_rate": 2.168888888888889e-06,
+      "loss": 0.0016,
+      "step": 4025
+    },
+    {
+      "epoch": 10.07,
+      "learning_rate": 2.1133333333333336e-06,
+      "loss": 0.0013,
+      "step": 4050
+    },
+    {
+      "epoch": 10.14,
+      "learning_rate": 2.057777777777778e-06,
+      "loss": 0.0014,
+      "step": 4075
+    },
+    {
+      "epoch": 10.2,
+      "learning_rate": 2.0022222222222225e-06,
+      "loss": 0.0013,
+      "step": 4100
+    },
+    {
+      "epoch": 10.26,
+      "learning_rate": 1.9466666666666665e-06,
+      "loss": 0.0011,
+      "step": 4125
+    },
+    {
+      "epoch": 10.32,
+      "learning_rate": 1.8911111111111114e-06,
+      "loss": 0.0012,
+      "step": 4150
+    },
+    {
+      "epoch": 10.39,
+      "learning_rate": 1.8355555555555557e-06,
+      "loss": 0.0016,
+      "step": 4175
+    },
+    {
+      "epoch": 10.45,
+      "learning_rate": 1.7800000000000001e-06,
+      "loss": 0.0014,
+      "step": 4200
+    },
+    {
+      "epoch": 10.51,
+      "learning_rate": 1.7244444444444448e-06,
+      "loss": 0.0015,
+      "step": 4225
+    },
+    {
+      "epoch": 10.57,
+      "learning_rate": 1.668888888888889e-06,
+      "loss": 0.0012,
+      "step": 4250
+    },
+    {
+      "epoch": 10.63,
+      "learning_rate": 1.6133333333333335e-06,
+      "loss": 0.0013,
+      "step": 4275
+    },
+    {
+      "epoch": 10.7,
+      "learning_rate": 1.5577777777777777e-06,
+      "loss": 0.0013,
+      "step": 4300
+    },
+    {
+      "epoch": 10.76,
+      "learning_rate": 1.5022222222222224e-06,
+      "loss": 0.0013,
+      "step": 4325
+    },
+    {
+      "epoch": 10.82,
+      "learning_rate": 1.4466666666666669e-06,
+      "loss": 0.0013,
+      "step": 4350
+    },
+    {
+      "epoch": 10.88,
+      "learning_rate": 1.3911111111111111e-06,
+      "loss": 0.0012,
+      "step": 4375
+    },
+    {
+      "epoch": 10.95,
+      "learning_rate": 1.3355555555555558e-06,
+      "loss": 0.0014,
+      "step": 4400
+    },
+    {
+      "epoch": 11.01,
+      "learning_rate": 1.28e-06,
+      "loss": 0.0014,
+      "step": 4425
+    },
+    {
+      "epoch": 11.07,
+      "learning_rate": 1.2244444444444445e-06,
+      "loss": 0.0014,
+      "step": 4450
+    },
+    {
+      "epoch": 11.13,
+      "learning_rate": 1.168888888888889e-06,
+      "loss": 0.001,
+      "step": 4475
+    },
+    {
+      "epoch": 11.19,
+      "learning_rate": 1.1133333333333334e-06,
+      "loss": 0.0008,
+      "step": 4500
+    },
+    {
+      "epoch": 11.26,
+      "learning_rate": 1.0577777777777779e-06,
+      "loss": 0.0014,
+      "step": 4525
+    },
+    {
+      "epoch": 11.32,
+      "learning_rate": 1.0022222222222223e-06,
+      "loss": 0.0012,
+      "step": 4550
+    },
+    {
+      "epoch": 11.38,
+      "learning_rate": 9.466666666666667e-07,
+      "loss": 0.0011,
+      "step": 4575
+    },
+    {
+      "epoch": 11.44,
+      "learning_rate": 8.911111111111112e-07,
+      "loss": 0.0015,
+      "step": 4600
+    },
+    {
+      "epoch": 11.5,
+      "learning_rate": 8.355555555555556e-07,
+      "loss": 0.0008,
+      "step": 4625
+    },
+    {
+      "epoch": 11.57,
+      "learning_rate": 7.8e-07,
+      "loss": 0.0013,
+      "step": 4650
+    },
+    {
+      "epoch": 11.63,
+      "learning_rate": 7.244444444444446e-07,
+      "loss": 0.0011,
+      "step": 4675
+    },
+    {
+      "epoch": 11.69,
+      "learning_rate": 6.68888888888889e-07,
+      "loss": 0.0012,
+      "step": 4700
+    },
+    {
+      "epoch": 11.75,
+      "learning_rate": 6.133333333333333e-07,
+      "loss": 0.0009,
+      "step": 4725
+    },
+    {
+      "epoch": 11.82,
+      "learning_rate": 5.577777777777779e-07,
+      "loss": 0.0011,
+      "step": 4750
+    },
+    {
+      "epoch": 11.88,
+      "learning_rate": 5.022222222222222e-07,
+      "loss": 0.001,
+      "step": 4775
+    },
+    {
+      "epoch": 11.94,
+      "learning_rate": 4.466666666666667e-07,
+      "loss": 0.0008,
+      "step": 4800
+    },
+    {
+      "epoch": 12.0,
+      "learning_rate": 3.9111111111111115e-07,
+      "loss": 0.0009,
+      "step": 4825
+    },
+    {
+      "epoch": 12.06,
+      "learning_rate": 3.3555555555555556e-07,
+      "loss": 0.0013,
+      "step": 4850
+    },
+    {
+      "epoch": 12.13,
+      "learning_rate": 2.8e-07,
+      "loss": 0.001,
+      "step": 4875
+    },
+    {
+      "epoch": 12.19,
+      "learning_rate": 2.2444444444444445e-07,
+      "loss": 0.0008,
+      "step": 4900
+    },
+    {
+      "epoch": 12.25,
+      "learning_rate": 1.6888888888888888e-07,
+      "loss": 0.0007,
+      "step": 4925
+    },
+    {
+      "epoch": 12.31,
+      "learning_rate": 1.1333333333333336e-07,
+      "loss": 0.0013,
+      "step": 4950
+    },
+    {
+      "epoch": 12.38,
+      "learning_rate": 5.777777777777778e-08,
+      "loss": 0.0011,
+      "step": 4975
+    },
+    {
+      "epoch": 12.44,
+      "learning_rate": 2.2222222222222225e-09,
+      "loss": 0.0008,
+      "step": 5000
+    },
+    {
+      "epoch": 12.44,
+      "eval_loss": 0.38012266159057617,
+      "eval_runtime": 1112.6036,
+      "eval_samples_per_second": 1.524,
+      "eval_steps_per_second": 0.095,
+      "eval_wer": 11.199851411589897,
+      "step": 5000
+    },
+    {
+      "epoch": 12.44,
+      "step": 5000,
+      "total_flos": 1.6322330164985856e+20,
+      "train_loss": 0.020854657278954983,
+      "train_runtime": 27142.1177,
+      "train_samples_per_second": 5.895,
+      "train_steps_per_second": 0.184
+    }
+  ],
+  "max_steps": 5000,
+  "num_train_epochs": 13,
+  "total_flos": 1.6322330164985856e+20,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:33ed7c38433905a1365c18790a32e9b80e38b4d99788b3a51c70edb3aecd69ff
-size 3643
+oid sha256:a3ebd5ec6b9b2945948080568a87ff00951d03abd4956eab0351149cc37e37fe
+size 3579