bezzam HF Staff commited on
Commit
bf2443c
·
verified ·
1 Parent(s): 9598146

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -111
README.md CHANGED
@@ -8,114 +8,4 @@ hardware: a100-large
8
  pinned: false
9
  ---
10
 
11
- # Transformers-library ASR Evaluation
12
-
13
- This folder contains evaluation scripts for ASR models supported by the 🤗 Transformers library.
14
-
15
- ## Supported Models
16
-
17
- | Script | Models |
18
- |--------|--------|
19
- | `run_whisper.sh` | OpenAI Whisper, Distil-Whisper, CrisperWhisper |
20
- | `run_wav2vec2.sh` | Wav2Vec2 |
21
- | `run_wav2vec2_conformer.sh` | Wav2Vec2 Conformer |
22
- | `run_hubert.sh` | HuBERT |
23
- | `run_data2vec.sh` | Data2Vec |
24
- | `run_mms.sh` | MMS |
25
- | `run_moonshine.sh` | Moonshine, Moonshine Streaming |
26
- | `run_voxtral.sh` | Voxtral Mini, Voxtral Small |
27
- | `run_voxtral_realtime.sh` | Voxtral Realtime |
28
- | `run_vibevoice.sh` | VibeVoice |
29
- | `run_glm_asr.sh` | GLM-ASR |
30
- | `run_granite.sh` | Granite Speech |
31
-
32
- ### Multilingual
33
-
34
- | Script | Models |
35
- |--------|--------|
36
- | `run_whisper_ml.sh` | OpenAI Whisper (multilingual) |
37
- | `run_voxtral_ml.sh` | Voxtral Mini, Voxtral Small |
38
- | `run_voxtral_realtime_ml.sh` | Voxtral Realtime |
39
-
40
- Multilingual scripts evaluate on FLEURS, MCV (Mozilla Common Voice), and MLS (Multilingual LibriSpeech) for German, French, Italian, Spanish, and Portuguese. They use `run_eval_ml.py` which applies language-specific normalization. By default, models auto-detect the language during inference as per the leaderboard convention. The argument `--language` can be used to force a specific language.
41
-
42
- ## Docker usage (recommended)
43
-
44
- From the **repository root**, build the Docker image:
45
-
46
- ```bash
47
- docker build -t open-asr-transformers -f transformers/Dockerfile .
48
- ```
49
-
50
- ### Run a specific script directly
51
-
52
- From the **repository root**, you can run a script without entering the container. The command below uses `--gpus` to expose all GPUs, mounts the local repo so scripts reflect latest changes, and mounts the HuggingFace cache for model downloads:
53
-
54
- ```bash
55
- docker run --gpus all \
56
- -v $(pwd):/app \
57
- -v $HF_HOME:/root/.cache/huggingface \
58
- open-asr-transformers run_whisper.sh
59
- ```
60
-
61
- Results are written to `transformers/results/` and are automatically persisted on the host since the repo is mounted.
62
-
63
- To select a specific GPU (e.g. GPU 1):
64
-
65
- ```bash
66
- docker run --gpus '"device=1"' \
67
- -v $(pwd):/app \
68
- -v $HF_HOME:/root/.cache/huggingface \
69
- open-asr-transformers run_whisper.sh
70
- ```
71
-
72
- ### Run interactively
73
-
74
- From the **repository root**, you can also enter the container to run interactively:
75
-
76
- ```bash
77
- docker run --gpus all -it \
78
- -v $(pwd):/app \
79
- -v $HF_HOME:/root/.cache/huggingface \
80
- open-asr-transformers -i
81
- ```
82
-
83
- This drops you into a bash shell inside `/app/transformers`. From there, run any evaluation script:
84
-
85
- ```bash
86
- # Evaluate all Whisper models
87
- bash run_whisper.sh
88
-
89
- # Evaluate Granite models
90
- bash run_granite.sh
91
-
92
- # Evaluate a single model/dataset manually
93
- python run_eval.py \
94
- --model_id=openai/whisper-large-v3-turbo \
95
- --dataset_path="hf-audio/open-asr-leaderboard" \
96
- --dataset="librispeech" \
97
- --split="test.clean" \
98
- --device=0 \
99
- --batch_size=64 \
100
- --max_eval_samples=-1
101
- ```
102
-
103
- ### Docker cheat sheet
104
-
105
- - Exit and stop a container, type `exit` or press `Ctrl+D`.
106
- - Detach from a container (without stopping): `Ctrl+P` then `Ctrl+Q`.
107
- - List running containers: `docker ps -a`.
108
- - Attach to a container: `docker attach <container_id>`
109
- - Delete a container: `docker rm <container_id>`
110
-
111
- ## Local Setup (without Docker)
112
-
113
- From the repository root:
114
-
115
- ```bash
116
- pip install -r requirements/requirements.txt
117
- pip install "mistral-common[audio]>=1.9.0" # only needed for Voxtral
118
- pip install peft # for Granite
119
- cd transformers
120
- bash run_whisper.sh
121
- ```
 
8
  pinned: false
9
  ---
10
 
11
+ # Dockerfile for evaluation Transformers models