Luigi/x-asr-zh-en-streaming-ntuml2021-ft-demo

X-ASR zh-en streaming zipformer2 — NTUML2021 encoder fine-tune (DEMONSTRATION)

⚠️ This is a research demonstration, not a production model. It exists to show that the X-ASR streaming zipformer2 transducer can be fine-tuned end-to-end and re-deployed to a Jetson Nano. It is a fine-tune of the base (non-punctuation) X-ASR checkpoint on a small, single-domain set (NTU "Machine Learning 2021" lectures), so its gains are in-domain adaptation — it is not a generally-better zh-TW model and not a drop-in replacement for the deployed X-ASR model. Read the numbers below with the caveats.

Part of the jetson-stt project (on-edge zh-TW/en STT for a 2-core Jetson Nano attendant).

What it is

• Base: GilgameshWind/X-ASR-zh-en base / non-punctuation checkpoint (streaming_exp/pretrained.pt, tokenizer lang_5000), a causal streaming zipformer2 transducer (6 stacks / 19 layers, dims 192·256·512·768·512·256, vocab 5000, 16 kHz / 80-d fbank).
• Fine-tune: encoder-only partial FT (147 M trainable; decoder/joiner/embedding frozen), 300 steps, causal/streaming regime, ~40 % retention (English + mainland-zh mixed every batch) on 200 NTUML2021 train clips. Trained on an NVIDIA GB10 (k2 built from source for CUDA 13 / Blackwell).
• Format: sherpa-onnx streaming int8 (encoder.int8 + decoder fp32 + joiner.int8), 960 ms variant (chunk 48 / left 256). Output is Simplified, no punctuation (the base model's space). Apply OpenCC s2twp downstream for Taiwan-Traditional output.

Measured on a real Jetson Nano gen1 (int8 streaming, 2 threads, vs the base)

metric (40-clip slices)	base	this FT
Taiwan zh-en CS — MER (NTUML2021, in-domain)	0.411	0.125 (−70 %)
English WER — clean held-out (LibriSpeech, not in FT data)	0.064	0.061 (unchanged — no forgetting)
RTF @ 2 threads	0.579	0.583 (unchanged — zero budget cost)

Honest caveats

• In-domain gain. Train/eval are disjoint NTUML2021 clips but the same ML-lecture domain. This measures domain adaptation, not a generic Taiwan-accent improvement (the project found no generic accent gap vs the Taiwan SOTA — see the repo's docs/PHASE0_RESULTS.md).
• Base, non-punctuation model. Weaker absolute accuracy than the deployed punctuation X-ASR. Use the deployed model for production; use this only to study fine-tuning.
• Mainland-zh retention not cleanly measured (held-out set unavailable at build time).
• Small data (200 clips); not robustness-tested on noisy/far-field audio.

Usage (sherpa-onnx)

import sherpa_onnx
rec = sherpa_onnx.OnlineRecognizer.from_transducer(
    tokens="tokens.txt", encoder="encoder.int8.onnx",
    decoder="decoder.onnx", joiner="joiner.int8.onnx",
    num_threads=2, provider="cpu", decoding_method="greedy_search")

License & provenance

Apache-2.0, following upstream X-ASR / icefall / k2 (k2-fsa). Fine-tuning data: NTUML2021 (ky552/ML2021_ASR_ST) + LibriSpeech (CC-BY) + FLEURS (CC-BY) retention. Recipe & full results: vieenrose/jetson-stt (docs/FINETUNE_RESULTS.md).