MostafaMaroof/wav2vec2-arabic-phoneme-asr

🎙️ Arabic Phoneme ASR v2 — wav2vec2-large-xlsr-53

Fine-tuned facebook/wav2vec2-large-xlsr-53 for phoneme-level Arabic speech recognition using CTC, trained on ~74k utterances from two corpora:

Corpus	Utterances	Notes
tunis-ai/arabic_speech_corpus	1,913	studio-recorded MSA
IqraEval/Iqra_train	71.4k + 2.6k dev	MSA/Quranic mispronunciation-detection corpus

---

⚠️ v2 is a different model from v1 — metrics are not comparable

v1 (trained on ASC only, PER 3.95%) used a label space that could not represent hamza or emphatic consonants: its cleaning step deleted the < glottal-stop token, and lowercasing merged ص↔س, ط↔ت, ض↔د, ظ↔ز, ح↔ه. v2 fixes both and adds ~38× more training data:

1. Hamza (<) preserved as a phoneme class
2. Case preserved — emphatics S T D Z H and emphatic vowels A I U remain distinct from their plain counterparts
3. Geminates preserved as tokens (bb, dd, …) — shadda is recoverable from the output
4. Scheme harmonisation — ASC's allophone digits and stress marks (i0, ii1', a') are mapped to IqraEval's simpler scheme (i, ii, a); rare inconsistent tokens folded (Ah→AH, SH→sh, TH→th, J→j, G→g), junk - marker dropped
5. Label source for IqraEval is phoneme_aug — the sequence actually spoken, including deliberately injected mispronunciations

The result is a harder task measured on harder data, evaluated with a richer label space. v2's 14.4% PER and v1's 3.95% PER measure different things; for pronunciation-scoring applications (makharij / tajweed), v2's label space is the usable one.

---

Model Description

This model transcribes Arabic speech directly into a sequence of phoneme tokens rather than graphemes or words. It is intended for pronunciation assessment, mispronunciation detection, linguistic analysis, and downstream tasks that benefit from sub-word acoustic representations.

The base model's convolutional feature encoder is frozen; only the transformer layers and a freshly initialized CTC head are fine-tuned.

---

Training Details

Parameter	Value
Base model	facebook/wav2vec2-large-xlsr-53
Datasets	ASC (phonetic) + IqraEval (phoneme_aug)
Sample rate	16 kHz (IqraEval mp3 resampled)
Max clip length	20 s (49 clips dropped)
Epochs	4
Effective batch size	16 (2 × 8 gradient accumulation)
Learning rate	1e-4, linear decay, 500 warmup steps
Precision	fp16
Hardware	1× NVIDIA T4 (Kaggle), ~8.9 h

Data Splits

Split	Samples	Source
Train	72,440	ASC train + ASC test + IqraEval train, minus holdout, minus >20 s clips
Validation	2,588	IqraEval dev split
Test	800	held out from the merged pool (seed 42)

Preprocessing

• Audio resampled to 16 kHz
• Punctuation removed from phoneme strings (hamza < preserved)
• Case preserved (emphatics remain distinct)
• Geminates preserved
• ASC allophone digits and stress marks stripped (i0' → i, UU1 → UU)
• Rare-token folding: Ah→AH, SH→sh, TH→th, J→j, G→g; - dropped; dist → sil
• Rows with empty labels removed
• Phonemes joined with | (word delimiter token) before tokenisation to prevent character-level splitting

---

Vocabulary

75 phoneme tokens plus |, [UNK], [PAD] (78 total in vocab.json; the tokenizer adds <s>/</s> on top).

$ $$ * ** < << A AA AH D DD E EE H HH I II S SS T TT U UU Z ZZ
^ ^^ a aa b bb d dd f ff g gg h hh i ii j jj k kk l ll m mm
n nn p pp q qq r rr s sh sil ss t th tt u uu v w ww x xx y yy z zz

Capitals are emphatic/pharyngealized variants (S=ص vs s=س, T=ط vs t=ت, D=ض, Z=ظ, H=ح vs h=ه, and emphatic vowels A I U). Doubled tokens are geminates (shadda). < is the glottal stop (hamza).

---

Evaluation Results

Held-out test set (800 samples, merged pool — includes IqraEval audio with injected mispronunciations):

Metric	Score
PER (Phoneme Error Rate)	14.42%
CER	9.50%
Test loss	0.271

On a 100-sample analysis subset with post-processing (trailing sil/$ stripped, 3+ repetition runs collapsed):

Metric	Score
Mean PER	14.41%
Median PER	11.54%
Exact matches (PER = 0)	14 / 100

Validation PER trajectory over 4 epochs: 18.2% → 16.0% → 15.1% → 14.5% → 14.1% → 13.8% → 13.7% → 13.6% (plateaued).

Note on interpreting PER: the test labels include deliberately injected mispronunciations from the IqraEval corpus, and the corresponding audio is largely synthesized to match. Free-decoding PER on this set is therefore a conservative estimate; performance on clean canonical recitation is higher. For pronunciation scoring, this model is best used with forced alignment / GOP-style posterior scoring against an expected phoneme sequence rather than free decoding.

---

Error Analysis

Error mass is dominated by short-vowel (haraka) confusions and boundary jitter — the hardest part of Arabic phoneme recognition. Consonant articulation is clean: no plain-consonant confusions of the س/ص, ت/ط, د/ض type dominate the top substitutions.

Top substitutions (reference → prediction, 100 test samples):

Reference	Predicted	Count
i	a	14×
u	a	13×
aa	a	13×
a	aa	9×
a	i	8×

Emphatic-pair confusions are present but minor (I↔A 8×, Z→D 4×, t→T 3× in 100 samples). The most common insertions and deletions are short a (boundary/epenthetic vowel jitter).

---

Usage

from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
import torch, re

model_path = "MostafaMaroof/wav2vec2-arabic-phoneme-asr"

processor = Wav2Vec2Processor.from_pretrained(model_path)
model = Wav2Vec2ForCTC.from_pretrained(model_path)
model.eval()

# audio_array: numpy array, 16 kHz
inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt", padding=True)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.decode(predicted_ids[0])
phonemes = transcription.replace("|", " ").strip()

# Recommended post-processing
_TRAIL = re.compile(r"(?:\s*(?:sil|\$))+$")

def postprocess(seq, max_run=2):
    seq = _TRAIL.sub("", seq).strip()          # strip boundary artifacts
    out, prev, count = [], None, 0
    for t in seq.split():                       # collapse 3+ repetition runs
        count = count + 1 if t == prev else 1
        if count <= max_run:
            out.append(t)
        prev = t
    return " ".join(out)

print(postprocess(phonemes))
# e.g. → "f ii h i nn a x A y r aa t H i s aa n"

---

Limitations

• Trained on MSA and Quranic-style recitation; dialectal speech is out of domain.
• A large share of IqraEval audio is TTS-generated; expect some domain gap on spontaneous natural speech.
• Short-vowel (haraka) distinctions are the dominant error class; long/short vowel decisions near word boundaries are least reliable.
• Greedy CTC decoding occasionally repeats a token at clip ends — apply the repetition-collapse post-processing above.
• The model outputs phoneme sequences (Halabi-style scheme), not Arabic orthography.

---

Citation

If you use this model, please cite the base model and datasets:

@misc{conneau2020unsupervised,
  title={Unsupervised Cross-lingual Representation Learning for Speech Recognition},
  author={Conneau, Alexis and others},
  year={2020},
  eprint={2006.13979},
  archivePrefix={arXiv}
}

---

License

Inherits the license of facebook/wav2vec2-large-xlsr-53. Please review the original model card before use in commercial applications.