anuj-inavlabs/kupe-asr-en

Streaming English ASR — early Phase‑1 checkpoint

A 670M streaming speech‑to‑text model: audio is turned into Mimi codec tokens and the Kupe‑SLM‑670M language backbone reads them and writes English text. Small, fast, real‑time.

Split	WER	CER
Test (held‑out, n=500)	10.3 %	4.9 %
Validation	11.2 %	5.4 %
Silence → empty output	100 %	—

⚠️ Early Phase‑1 checkpoint. This is a first sanity‑milestone model trained on ~2.6k h to prove the pipeline and architecture. It is not the final release — quality will improve substantially with more data, augmentation, and epochs in later phases.

---

Overview

• Task: English automatic speech recognition (streaming + full‑file).
• How it works: speech → Mimi neural audio codec (8 codebooks @ 12.5 Hz) → the 8 codebooks are summed into one embedding per 80 ms frame → fed to Kupe‑SLM‑670M as inputs_embeds → the model decodes English text. No separate acoustic encoder.
• Streaming: emits turn signals from the model's own end‑of‑sequence probability — PRE_HIT_LLM (prefetch your LLM) and END_OF_SPEECH (commit final).
• Anti‑hallucination: trained with 5 % silence/ambient clips labelled "", so silence decodes to nothing (measured 100 %).

Results in context

Production English ASR on much larger corpora sits around 4–6 % WER. 10.3 % from a Phase‑1 670M model on 2.6k h is a strong early result and already usable for voice agents. Live WER during training fell 41 % → 18 % → 12.8 % → 10.3 %.

Training data (~2.6k h English, encoded)

Source	Hours	Clips
LibriSpeech (clean+other)	599	172,657
People's Speech (CC‑BY)	2,100	543,069
Svarah (Indian‑accented EN)	8.6	5,227
Silence / ambient (labelled "")	35	36,315
Total encoded	~2,606	716,968

24 kHz mono, Mimi‑encoded (8 codebooks). Splits: train ~2,733 h · val 5.1 h · test 5.0 h.

Training setup (1× H100 80 GB)

Method	full fine‑tune (all params), causal‑LM cross‑entropy
Effective batch	128 (micro‑batch 32 × grad‑accum 4)
LR / schedule	2e‑4, cosine, 3 % warmup
Epochs / steps	2 / 11,158
Precision	bf16 + TF32, fused AdamW
Regularisation	SpecAugment (time masking)
Wall‑clock	~3.5 h · final eval_loss 0.26

Tokenizer efficiency (fertility)

The 131,072‑vocabulary tokenizer is very efficient — low fertility (tokens/word) means shorter targets, faster decoding, lower CER.

Language	Fertility (tok/word)	Round‑trip
English	1.29	100 %
Hindi	1.63	100 %
Gujarati	1.60	100 %
Punjabi	1.47	100 %
Bengali	1.77	100 %
Marathi	1.93	100 %
Telugu / Kannada / Tamil / Odia	2.1–2.3	100 %
Malayalam	3.12	100 %

Multilingual outlook (this checkpoint is EN‑only)

The backbone was pre‑trained on English + 11 Indic languages; a probe found it already generates coherent Hindi & Gujarati and has 100 % tokenizer round‑trip across all 11 scripts. So extending to Hindi/Gujarati/… is mainly a matter of more data on the same pipeline — the model side is ready.

Files

• kupe-lm/ — the Kupe‑SLM‑670M backbone (weights + tokenizer).
• frontend.pt — audio frontend (codebook embeddings + projector).
• asr_config.json — frontend mode (per_frame_sum), codebooks (8), embed dim.

Inference

Requires transformers==5.4.0 and the kupe-asr-en code.

from kupe_asr_en.modeling.asr_model import LummaASR   # loader class name; loads Kupe-SLM
model, tok = LummaASR.load("anuj-inavlabs/kupe-asr-en", device="cuda")

Streaming from a wav (turn flags):

from kupe_asr_en.stream import transcribe_file
for ev in transcribe_file("model_dir", "audio.wav"):
    print(ev.type, ev.text)   # PARTIAL / PRE_HIT_LLM / END_OF_SPEECH

Live mic / Gradio demo (file + realtime, with RTF):

python scripts/06_mic_stream.py --model-dir <this repo>     # terminal mic
python scripts/10_gradio_kaggle.py                          # web UI (file + live mic)

Related repos

• Data: anuj-inavlabs/kupe-asr-en-data
• Runs: anuj-inavlabs/kupe-asr-en-runs
• Code: https://github.com/iNavLabsResearch/kupe-asr-en

License & credits

Speech‑LM backbone weights are used under their upstream license; audio codec: kyutai/mimi. Data: LibriSpeech, MLCommons People's Speech (CC‑BY), AI4Bharat Svarah. Released as other. Built by iNavLabs.