thunderboltc/whisper-small-santali-sanlish

Whisper Small — Santali ASR (Custom Transliteration / Sanlish)

Fine-tuned version of openai/whisper-small for Automatic Speech Recognition (ASR) of the Santali language, using a custom Roman-script transliteration scheme called Sanlish.

This model is part of a thesis research pipeline for low-resource Santali speech processing, targeting the task of Santali Speech → Sanlish Transcription → Bangla Translation.

---

Model Details

Model Description

• Developed by: thunderboltc
• Model type: Whisper Small (Encoder-Decoder Transformer, Seq2Seq)
• Language: Santali (sat) — transcribed in custom Roman transliteration (Sanlish)
• Base model: openai/whisper-small
• Parameters: 241,734,912 (~242M)
• Fine-tuned for: Low-resource ASR — Santali speech to Sanlish text
• License: MIT

What is Sanlish?

Sanlish is a custom Roman-script transliteration of Santali designed to bridge the gap between Santali phonetics and Latin-alphabet-based ASR systems. It was developed as part of this thesis to serve as an intermediate representation between Santali audio and Bangla translation.

---

Training Details

Training Data

• Dataset: 502 Santali audio recordings with parallel Sanlish transliteration annotations
• Collected by: thunderboltc (thesis dataset, not yet publicly released)
• Split:

Split	Size	Percentage
Train	401	80%
Validation	50	10%
Test	51	10%

• Audio format: WAV, 16kHz mono
• Text target: Custom Roman transliteration (Sanlish)

Training Procedure

Preprocessing

• Audio resampled to 16,000 Hz
• Log-mel spectrogram features extracted using WhisperFeatureExtractor (80 mel bins, 3000 frames)
• Text tokenized using WhisperTokenizer with language=english, task=transcribe
• forced_decoder_ids and suppress_tokens disabled to allow custom vocabulary transcription

Training Hyperparameters

Parameter	Value
Epochs	25
Learning Rate	1e-5
Batch Size (per device)	8
Gradient Accumulation Steps	2 (effective batch = 16)
Warmup Steps	50
Precision	fp16 mixed precision
Evaluation Strategy	Per epoch
Best Model Selection	Lowest validation WER
Hardware	NVIDIA Tesla T4 (Google Colab)
Training Duration	~73 minutes

Best Checkpoint

Epoch 13 was selected as the best checkpoint based on lowest validation WER (53.41%).

---

Evaluation

Results on Test Set (51 utterances)

Metric	Value
Word Error Rate (WER)	56.55%
Character Error Rate (CER)	13.31%

These results are consistent with expectations for a low-resource ASR system trained on ~400 utterances. The large gap between WER and CER reflects that the model captures phonetic character structure well, even when full word sequences deviate from the reference.

Sample Predictions

Reference	Prediction
onde alom chala a	onde alom chala a ✓
tehindo adi raashkaien yaikaoda	tehindo adi reskaien yaikaoda
gapa shetah ale saote baskiyago meh	gapa setah ale saote baskiyago meh
shetay bhuh da	shetay bhuheda

---

How to Get Started

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

# Load model and processor
processor = WhisperProcessor.from_pretrained("thunderboltc/whisper-small-santali-sanlish")
model = WhisperForConditionalGeneration.from_pretrained("thunderboltc/whisper-small-santali-sanlish")

# Load audio (must be 16kHz)
audio, sr = librosa.load("your_santali_audio.wav", sr=16000)

# Preprocess
inputs = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt")

# Generate transcription
with torch.no_grad():
    predicted_ids = model.generate(inputs.input_features)

# Decode
transcription = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)
print("Sanlish transcription:", transcription[0])

---

Intended Use

Direct Use

Transcribing Santali speech into Sanlish (custom Roman transliteration) for downstream NLP tasks.

Downstream Use

This model is designed to feed into a Machine Translation (MT) model (NLLB-200) for end-to-end Santali Speech → Bangla Text translation.

Out-of-Scope Use

• Not suitable for general-purpose ASR in other languages
• Not recommended for standard Santali Ol Chiki script transcription
• Performance may degrade on speaker accents or recording conditions not represented in the 502-utterance training set

---

Limitations & Bias

• Trained on only 502 utterances — a very small dataset by ASR standards
• May not generalize well to unseen speakers or recording environments
• Sanlish transliteration scheme has inherent inconsistencies (word boundary variation, hyphenation) which inflate WER beyond true phonetic error rate
• No speaker diversity information available for the training set

---

Citation

If you use this model in your research, please cite:

@misc{thunderboltc2025whisper-santali,
  author = {thunderboltc},
  title = {Whisper Small Fine-tuned on Santali Custom Transliteration (Sanlish)},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/thunderboltc/whisper-small-santali-sanlish}
}

---

Environmental Impact

• Hardware: NVIDIA Tesla T4
• Cloud Provider: Google Colab
• Hours used: ~1.25 hours
• Carbon emissions: Estimated using the ML Impact Calculator