README.md

---
language:
- ar
license: apache-2.0
base_model: facebook/wav2vec2-large-xlsr-53
datasets:
- tunis-ai/arabic_speech_corpus
- IqraEval/Iqra_train
tags:
- automatic-speech-recognition
- arabic
- phoneme
- ctc
- wav2vec2
- pronunciation
- quran
metrics:
- wer
model-index:
- name: wav2vec2-large-xlsr-53-arabic-phoneme
  results:
  - task:
      type: automatic-speech-recognition
      name: Automatic Speech Recognition
    dataset:
      name: ASC + IqraEval (merged, held-out test)
      type: IqraEval/Iqra_train
      split: test
    metrics:
    - type: wer
      value: 0.1442
      name: PER (Phoneme Error Rate)
    - type: cer
      value: 0.095
      name: CER
---

# 🎙️ Arabic Phoneme ASR v2 — wav2vec2-large-xlsr-53

Fine-tuned [`facebook/wav2vec2-large-xlsr-53`](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) for **phoneme-level Arabic speech recognition** using CTC, trained on ~74k utterances from two corpora:

| Corpus | Utterances | Notes |
|---|---|---|
| [`tunis-ai/arabic_speech_corpus`](https://huggingface.co/datasets/tunis-ai/arabic_speech_corpus) | 1,913 | studio-recorded MSA |
| [`IqraEval/Iqra_train`](https://huggingface.co/datasets/IqraEval/Iqra_train) | 71.4k + 2.6k dev | MSA/Quranic mispronunciation-detection corpus |

---

## ⚠️ v2 is a different model from v1 — metrics are not comparable

v1 (trained on ASC only, PER 3.95%) used a label space that **could not represent hamza or emphatic consonants**: its cleaning step deleted the `<` glottal-stop token, and lowercasing merged ص↔س, ط↔ت, ض↔د, ظ↔ز, ح↔ه. v2 fixes both and adds ~38× more training data:

1. **Hamza (`<`) preserved** as a phoneme class
2. **Case preserved** — emphatics `S T D Z H` and emphatic vowels `A I U` remain distinct from their plain counterparts
3. **Geminates preserved** as tokens (`bb`, `dd`, …) — shadda is recoverable from the output
4. **Scheme harmonisation** — ASC's allophone digits and stress marks (`i0`, `ii1'`, `a'`) are mapped to IqraEval's simpler scheme (`i`, `ii`, `a`); rare inconsistent tokens folded (`Ah→AH`, `SH→sh`, `TH→th`, `J→j`, `G→g`), junk `-` marker dropped
5. Label source for IqraEval is `phoneme_aug` — the sequence actually spoken, including deliberately injected mispronunciations

The result is a harder task measured on harder data, evaluated with a richer label space. v2's 14.4% PER and v1's 3.95% PER measure different things; for pronunciation-scoring applications (makharij / tajweed), v2's label space is the usable one.

---

## Model Description

This model transcribes Arabic speech directly into a sequence of phoneme tokens rather than graphemes or words. It is intended for pronunciation assessment, mispronunciation detection, linguistic analysis, and downstream tasks that benefit from sub-word acoustic representations.

The base model's convolutional feature encoder is frozen; only the transformer layers and a freshly initialized CTC head are fine-tuned.

---

## Training Details

| Parameter | Value |
|---|---|
| Base model | `facebook/wav2vec2-large-xlsr-53` |
| Datasets | ASC (`phonetic`) + IqraEval (`phoneme_aug`) |
| Sample rate | 16 kHz (IqraEval mp3 resampled) |
| Max clip length | 20 s (49 clips dropped) |
| Epochs | 4 |
| Effective batch size | 16 (2 × 8 gradient accumulation) |
| Learning rate | 1e-4, linear decay, 500 warmup steps |
| Precision | fp16 |
| Hardware | 1× NVIDIA T4 (Kaggle), ~8.9 h |

### Data Splits

| Split | Samples | Source |
|---|---|---|
| Train | 72,440 | ASC train + ASC test + IqraEval train, minus holdout, minus >20 s clips |
| Validation | 2,588 | IqraEval `dev` split |
| Test | 800 | held out from the merged pool (seed 42) |

### Preprocessing

- Audio resampled to 16 kHz
- Punctuation removed from phoneme strings (hamza `<` **preserved**)
- Case **preserved** (emphatics remain distinct)
- Geminates **preserved**
- ASC allophone digits and stress marks stripped (`i0' → i`, `UU1 → UU`)
- Rare-token folding: `Ah→AH`, `SH→sh`, `TH→th`, `J→j`, `G→g`; `-` dropped; `dist → sil`
- Rows with empty labels removed
- Phonemes joined with `|` (word delimiter token) before tokenisation to prevent character-level splitting

---

## Vocabulary

75 phoneme tokens plus `|`, `[UNK]`, `[PAD]` (78 total in `vocab.json`; the tokenizer adds `<s>`/`</s>` on top).

```
$ $$ * ** < << A AA AH D DD E EE H HH I II S SS T TT U UU Z ZZ
^ ^^ a aa b bb d dd f ff g gg h hh i ii j jj k kk l ll m mm
n nn p pp q qq r rr s sh sil ss t th tt u uu v w ww x xx y yy z zz
```

Capitals are emphatic/pharyngealized variants (`S`=ص vs `s`=س, `T`=ط vs `t`=ت, `D`=ض, `Z`=ظ, `H`=ح vs `h`=ه, and emphatic vowels `A I U`). Doubled tokens are geminates (shadda). `<` is the glottal stop (hamza).

---

## Evaluation Results

Held-out test set (800 samples, merged pool — includes IqraEval audio with injected mispronunciations):

| Metric | Score |
|---|---|
| **PER** (Phoneme Error Rate) | **14.42%** |
| **CER** | **9.50%** |
| Test loss | 0.271 |

On a 100-sample analysis subset with post-processing (trailing `sil`/`$` stripped, 3+ repetition runs collapsed):

| Metric | Score |
|---|---|
| Mean PER | 14.41% |
| Median PER | 11.54% |
| Exact matches (PER = 0) | 14 / 100 |

Validation PER trajectory over 4 epochs: 18.2% → 16.0% → 15.1% → 14.5% → 14.1% → 13.8% → 13.7% → 13.6% (plateaued).

**Note on interpreting PER:** the test labels include deliberately injected mispronunciations from the IqraEval corpus, and the corresponding audio is largely synthesized to match. Free-decoding PER on this set is therefore a conservative estimate; performance on clean canonical recitation is higher. For pronunciation scoring, this model is best used with forced alignment / GOP-style posterior scoring against an expected phoneme sequence rather than free decoding.

---

## Error Analysis

Error mass is dominated by **short-vowel (haraka) confusions and boundary jitter** — the hardest part of Arabic phoneme recognition. Consonant articulation is clean: no plain-consonant confusions of the س/ص, ت/ط, د/ض type dominate the top substitutions.

**Top substitutions (reference → prediction, 100 test samples):**

| Reference | Predicted | Count |
|---|---|---|
| `i` | `a` | 14× |
| `u` | `a` | 13× |
| `aa` | `a` | 13× |
| `a` | `aa` | 9× |
| `a` | `i` | 8× |

Emphatic-pair confusions are present but minor (`I↔A` 8×, `Z→D` 4×, `t→T` 3× in 100 samples). The most common insertions and deletions are short `a` (boundary/epenthetic vowel jitter).

---

## Usage

```python
from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
import torch, re

model_path = "MostafaMaroof/wav2vec2-arabic-phoneme-asr"

processor = Wav2Vec2Processor.from_pretrained(model_path)
model = Wav2Vec2ForCTC.from_pretrained(model_path)
model.eval()

# audio_array: numpy array, 16 kHz
inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.decode(predicted_ids[0])
phonemes = transcription.replace("|", " ").strip()

# Recommended post-processing
_TRAIL = re.compile(r"(?:\s*(?:sil|\$))+$")

def postprocess(seq, max_run=2):
    seq = _TRAIL.sub("", seq).strip()          # strip boundary artifacts
    out, prev, count = [], None, 0
    for t in seq.split():                       # collapse 3+ repetition runs
        count = count + 1 if t == prev else 1
        if count <= max_run:
            out.append(t)
        prev = t
    return " ".join(out)

print(postprocess(phonemes))
# e.g. → "f ii h i nn a x A y r aa t H i s aa n"
```

---

## Limitations

- Trained on MSA and Quranic-style recitation; dialectal speech is out of domain.
- A large share of IqraEval audio is TTS-generated; expect some domain gap on spontaneous natural speech.
- Short-vowel (haraka) distinctions are the dominant error class; long/short vowel decisions near word boundaries are least reliable.
- Greedy CTC decoding occasionally repeats a token at clip ends — apply the repetition-collapse post-processing above.
- The model outputs phoneme sequences (Halabi-style scheme), not Arabic orthography.

---

## Citation

If you use this model, please cite the base model and datasets:

```bibtex
@misc{conneau2020unsupervised,
  title={Unsupervised Cross-lingual Representation Learning for Speech Recognition},
  author={Conneau, Alexis and others},
  year={2020},
  eprint={2006.13979},
  archivePrefix={arXiv}
}
```

---

## License

Inherits the license of `facebook/wav2vec2-large-xlsr-53`. Please review the [original model card](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) before use in commercial applications.