fujie/espnet_asr_csj_cejc_pron_comp_cbs_ctc_120300_hop132

Streaming Japanese ASR that also marks fillers and repairs, trained on CSJ + CEJC by Shinya Fujie (Fujie Lab, Chiba Institute of Technology).

CTC-only head — no decoder, greedy decoding. This is the composite tokens (ア+F / ン+D) variant. The auxiliary information is fused into the token itself: a filler kana is recognized as ア+F, a repair kana as ン+D. Strip the +F / +D suffix to recover the plain transcript.

Sibling models: the same three encodings with a CTC head (aux / comp / span) and with an RNN-Transducer head.

Usage

pip install "fujielab-asr>=0.2.0"

import numpy as np, soundfile as sf
from fujielab.asr.espnet_ext.espnet2.bin.asr_ctc_inference_cbs import Speech2TextCTC

s2t = Speech2TextCTC.from_pretrained("fujie/espnet_asr_csj_cejc_pron_comp_cbs_ctc_120300_hop132", streaming=True)

audio, fs = sf.read("utterance.wav")   # 16 kHz mono
chunk = int(16000 * 0.1)               # 100 ms
n, result = len(audio), None
for i in range(0, n, chunk):
    c = audio[i:i + chunk]
    is_final = i + chunk >= n          # flag the last chunk even when it is full
    if len(c) < chunk:
        c = np.pad(c, (0, chunk - len(c)))
    r = s2t.streaming_decode(c, is_final=is_final)
    if r:
        result = r[0]
print(" ".join(result.tokens))

Evaluation

17 evaluation sets (CEJC 14 + CSJ 3), 23,500 utterances, greedy CTC decoding with valid.cer_ctc.ave_10best.pth.

CER is computed with all auxiliary information stripped from both reference and hypothesis, so every encoding is measured against the same target.

	CER %	F: P / R / F1	D: P / R / F1
this model	17.64	77.5 / 83.6 / 80.4	55.4 / 36.9 / 44.3

By corpus: CEJC 24.71 % / CSJ 6.21 % CER.

F = filler (フィラー), D = repair / disfluent restart (言い直し). A detection counts as correct when the predicted span overlaps a reference span of the same type within ±1 token.

CTC vs. RNN-Transducer

A sibling model with the identical encoder, data, effective batch (32 M), schedule (50 × 1400 steps, lr 0.0035, warmup 2500) and evaluation exists with an RNN-Transducer head — only the head differs, so the gap is attributable to it:

head	CER %
CTC (this model)	17.64
RNN-T (..._comp_cbs_transducer_120300_hop132)	16.25
difference	+1.39

The RNN-T head wins on all three encodings (+0.33 to +1.39 points, each significant under a paired bootstrap over utterances), but the margin varies fourfold by encoding, so there is no single "RNN-T is N points better" figure.

Auxiliary-label quality barely depends on the head: across all six models F F1 spans 80.4–82.0 and D F1 spans 41.6–45.5. What does change is the precision/recall balance — the CTC models lean toward recall, the RNN-T models are balanced — which is the more useful basis for choosing between them.

Pick CTC when you want a smaller, faster, decoder-free model and can accept ~0.3–1.4 points more CER; pick RNN-T for the best transcript.

Training

• Encoder: Contextual Block Conformer, 12 blocks, d=256, 4 heads, macaron + conv module (kernel 15). Left context 12 / main block 3 / look-ahead 0, block_size 15, hop_size 3 — causal at block granularity.
• Head: CTC only (ctc_weight 1.0, no decoder).
• Frontend: 80-dim log-mel, hop_length 132 samples (8.25 ms) at 16 kHz, ×4 subsampling → one encoder frame per 33 ms.
• Data: CSJ + CEJC, kana (pronunciation) tokens, CSJ-aligned filler criterion (short vowel fillers えー / まあ / あー / ん also count as F).
• Schedule: 50 epochs × 1400 steps, Adam lr 0.0035, warmup 2500, effective batch 32 M bins (16 M × 2 × H100). Peak memory 55 GB.

Note on positional encoding and long audio

The encoder adds an absolute sinusoidal positional encoding indexed from the start of the stream, and uses plain (not relative) self-attention. Training utterances reach 19.9 s at most (498 encoder frames), so a long continuous stream pushes the position index outside the trained range. Segment the audio (e.g. by VAD) and let the recognizer reset per segment.

Citing ESPnet

@inproceedings{watanabe2018espnet,
  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi
          and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann
          and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala
          and Tsubasa Ochiai},
  title={{ESPnet}: End-to-End Speech Processing Toolkit},
  year={2018},
  booktitle={Proceedings of Interspeech},
  pages={2207--2211},
  doi={10.21437/Interspeech.2018-1456}
}