Jianshu001/wavlm-phoneme-scorer

WavLM Phoneme Scorer

Fine-tuned WavLM-Large backbone + MLP scoring head for phoneme-level English pronunciation assessment, trained on 11K children's speech samples.

Given a reference text and an audio recording, this model identifies which phonemes are pronounced correctly and which have errors.

Quick Start

Install

pip install torch torchaudio transformers g2p-en huggingface_hub

Python API

from pipeline_v2 import PronunciationAssessorV2

# Auto-download model from HuggingFace
assessor = PronunciationAssessorV2.from_pretrained()

result = assessor.assess("audio.mp3", "Hello, Peter.")

print(result["overall_score"])        # 85.2
print(result["n_errors"])             # 0
for word in result["words"]:
    print(f"{word['word']}: score={word['score']}")
    for ph in word["phonemes"]:
        err = " <- ERROR" if ph["error"] else ""
        print(f"  /{ph['phone']}/  score={ph['score']}  pherr={ph['pherr_prob']:.2f}{err}")

CLI

# Model downloads automatically on first run
python pipeline_v2.py --audio audio.mp3 --text "Hello, Peter."

Output: ```

Text: "Hello, Peter." Overall Score: 85.2/100 (errors: 0/8)

✓ Hello score= 87.7 errors=0/4 /hh / score= 98.6 GOP= -0.97 pherr=0.05 /ah / score= 73.1 GOP= -7.40 pherr=0.43 /l / score= 88.6 GOP= +4.00 pherr=0.29 /ow / score= 90.7 GOP= -6.05 pherr=0.13

✓ Peter score= 82.6 errors=0/4 /p / score= 95.7 GOP= +5.40 pherr=0.08 /iy / score= 90.2 GOP= +3.70 pherr=0.12 /t / score= 72.5 GOP= +0.50 pherr=0.55 /er / score= 71.8 GOP= -1.40 pherr=0.61

### Download Model Manually

```bash
# Via huggingface-cli
huggingface-cli download Jianshu001/wavlm-phoneme-scorer wavlm_finetuned.pt --local-dir .

# Via Python
from huggingface_hub import hf_hub_download
hf_hub_download(repo_id="Jianshu001/wavlm-phoneme-scorer", filename="wavlm_finetuned.pt", local_dir=".")

# Then use with local path
assessor = PronunciationAssessorV2(checkpoint_path="wavlm_finetuned.pt")

Architecture

Reference Text ──→ G2P ──→ Expected phoneme sequence
                                    │
Audio ──→ CTC model ──→ Viterbi Forced Alignment ──→ Frame segments
  │                                                       │
  └──→ WavLM-Large (fine-tuned) ──→ Hidden states ──→ Pool per segment
                                                          │
                                                    + phone embedding (32d)
                                                    + GOP score (1d)
                                                    + n_frames (1d)
                                                          │
                                                    MLP (1058 → 512 → 512 → 256)
                                                    ├── score_head → phoneme score (0-100)
                                                    └── pherr_head → error probability (0-1)

• Backbone: WavLM-Large with top 6 transformer layers fine-tuned (76.6M trainable / 316.4M total)
• Alignment: facebook/wav2vec2-xlsr-53-espeak-cv-ft (frozen, CTC-based Viterbi forced alignment)
• Scoring head: MLP with BatchNorm, GELU, Dropout, multi-task (score regression + error classification)

Performance

Evaluated on test set (8062 phonemes from 1727 audio files, children's speech):

Metric	GOP Baseline (v1.0)	This Model
Phoneme Error AUC-ROC	0.738	0.870
Phoneme Error F1	0.476	0.595
Phoneme Error Precision	0.379	0.592
Phoneme Error Recall	0.638	0.598
Phone Score Pearson	0.372	0.645
Phone Score MAE	27.44	16.47

Training Data

• 11,601 audio recordings of English learners (children)
• 53,926 phonemes with professional human evaluation labels
• Labels include per-phoneme scores (0-100) and error flags (pherr 0/1)

Training Details

• Fine-tuned top 6 of 24 WavLM transformer layers
• Differential learning rate: backbone 1e-5, head 5e-4
• AdamW optimizer, cosine annealing, early stopping (patience=8)
• Gradient accumulation (4 steps), gradient clipping (max_norm=1.0)
• Batch size 64, trained for 24 epochs (early stopped)
• Multi-task loss: MSE/100 (score) + BCE with pos_weight=5.2 (error)
• Train/Val/Test split by audio file: 40K/5K/8K phonemes

Files

File	Description	Size
wavlm_finetuned.pt	Full checkpoint (backbone + head state dict)	1.2GB
pipeline_v2.py	Inference pipeline with from_pretrained() support	18KB
finetune_wavlm.py	Training script (reproducing the fine-tuning)	25KB

Full Repository

See the complete project (data, evaluation, all experiments) at: github.com/Jianshu-She/Voice-correction (branch: feature/wavlm-pipeline)