You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Real-Time Streaming ASR — Kazakh / Russian / English (FastConformer-TDT)

A cache-aware streaming speech-to-text model for Kazakh, Russian, and English, packaged as a NVIDIA NeMo .nemo checkpoint. Built for real-time / server-side transcription (WebSocket, low-latency streaming) while staying balanced across all three languages — no single language is favored.

Task: automatic speech recognition / speech-to-text (streaming and offline).
Languages: Kazakh (kk), Russian (ru), English (en).
Architecture: 115M FastConformer-large (d_model 512, 17 layers, ×8 subsampling), NeMo EncDecHybridRNNTCTCBPEModel, TDT decoder (token-and-duration transducer, durations 0–4) + auxiliary CTC (0.3).
Streaming: cache-aware, att_context_style: chunked_limited, causal convolutions + causal downsampling. Four selectable latency profiles share one set of weights: [[70,13],[70,6],[70,1],[70,0]] ≈ 1040 / 480 / 80 / 0 ms right-context lookahead.
Tokenizer: fresh 4096-piece trilingual BPE, balanced (equal lines per language) so Latin/Cyrillic/Kazakh graphemes and digits are all well covered.
Format: NeMo .nemo. Input audio: mono 16 kHz.

This is a cache-aware streaming checkpoint (not a full-context offline-only model, not TTS, not diarization).

Evaluation

Held-out WER (%), uniform Whisper text normalization, corpus WER. Evaluation sets: FLEURS test (500 seeded utts / language) for kk/ru/en, and ISSAI KSC2 test (1000 seeded utts, the canonical in-domain Kazakh benchmark). Lower is better. Numbers are measured on these sets, not taken from a model card.

Decoding profile	Lookahead	KK (FLEURS)	RU (FLEURS)	EN (FLEURS)	KK (KSC2)	FLEURS macro	RTF
Offline (full-context attn)	—	13.50	14.95	16.31	12.69	14.92	~0.009
Streaming `[70,13]`	~1040 ms	13.90	14.94	16.39	12.89	15.08	~0.015
Streaming `[70,6]`	~480 ms	14.34	16.20	17.34	13.75	15.96	~0.016
Streaming `[70,1]`	~80 ms	15.89	18.73	20.03	15.71	18.22	~0.037

Reading the table

Balanced across languages. FLEURS spread (max−min) is ~2.5–2.9 WER at every profile; RU and EN are not undersampled or sacrificed for KK.
Real-time at production latency. At [70,13] (~1 s lookahead) the FLEURS macro is 15.08 — matching the offline number, i.e. streaming costs almost nothing at 1 s. RTF ≈ 0.015 (≈ 65× faster than real-time on a single GPU).
Strong on canonical Kazakh. KSC2 is 12.69 offline / 12.89 streaming.
Graceful degradation all the way down to an 80 ms lookahead for ultra-low-latency use.

Per-profile CER and per-set counts are in metadata/eval.json.

Files

fc_tdt_streaming_kk_ru_en.nemo — NeMo streaming ASR checkpoint.
onnx/ — end-to-end ONNX export (cache-aware encoder + TDT decoder_joint, opset 17) with tokenizer + config for ONNX Runtime / Triton serving. See onnx/README.md.
metadata/eval.json — full evaluation summary (WER/CER/RTF, all profiles).
examples/offline_asr.py — batch/file transcription.
examples/streaming_infer.py — cache-aware streaming transcription with a selectable latency profile.

Install

Use an environment with NVIDIA NeMo ASR support and CUDA-enabled PyTorch.

pip install "nemo_toolkit[asr]" huggingface_hub soundfile

Download

from huggingface_hub import hf_hub_download

model_path = hf_hub_download(
    repo_id="nur-dev/realtime-streaming-asr-kk-ru-en",
    filename="fc_tdt_streaming_kk_ru_en.nemo",
)

Quick start (offline)

import nemo.collections.asr as nemo_asr

model = nemo_asr.models.ASRModel.restore_from(model_path).eval()
print(model.transcribe(["audio_16k_mono.wav"])[0].text)

Streaming latency profile

The model ships four cache-aware profiles in one checkpoint. Select the latency/accuracy trade-off before decoding:

# ~1040 ms lookahead (best accuracy) ... ~80 ms (lowest latency)
model.encoder.set_default_att_context_size([70, 13])   # or [70,6], [70,1]

For true cache-aware streaming inference, use NeMo's cache_aware_streaming utilities (see examples/streaming_infer.py).

Training

8× NVIDIA L40, pure DDP, bf16-mixed, 150k steps.
~2413 h trilingual training audio, balanced by hours (kk 33.4 % / ru 33.2 % / en 33.4 %).
Encoder warm-started from a cache-aware streaming trilingual base; TDT decoder, joint, CTC and the 4096 BPE tokenizer trained fresh for this vocabulary.
Best blended validation WER: 0.0945.

Intended use & limitations

Intended for real-time / server-side transcription of kk/ru/en speech at 16 kHz.
Best accuracy at [70,13]; use lower profiles when latency matters more than WER.
English at the tightest 80 ms profile (20.0) is the weakest operating point; prefer [70,13]/[70,6] for English-heavy traffic.
Numbers do not represent written-form/ITN output; RU numerals follow the training convention.

License

Released under CC BY-NC 4.0 (non-commercial). For commercial licensing or inquiries, contact the author.

Contact

Author: Nurgali Kadyrbek — kadyrbeknurgali@gmail.com. For commercial licensing, connect on LinkedIn.

Downloads last month: 4