CSJ pron span — CTC ASR with in-token auxiliary information

Streaming CTC model for Japanese (kana) that recognizes content together with filler (F) and repair (D) markers encoded directly in the tokens: span range markers (<F> あの </F> / <D> ... </D>). No separate auxiliary head — a plain CTC model.

Use with fujielab-asr (>= 0.1.6).

Architecture

Encoder: Contextual Block Streaming Transformer (left context 12 / main block 3 / look-ahead 0 → block_size=15, hop_size=3, look_ahead=0), output 256, 12 blocks.
CTC-only (no decoder); word-level kana tokens with <F> あ </F> markers.
Frontend hop_length 132 (8.25 ms @ 16 kHz).

Results (CSJ eval1-3, 4250 utt, 10-best averaged)

content CER (markers removed): 6.76%
filler (F) detection F1: 93.1
repair (D) detection F1: 43.8 (F/D scored after normalizing to span representation and aligning hyp↔gold, tolerating small positional/content differences.)

This is one of three encodings (multitask / composite / span) of the same filler/repair information, trained on the same utterances with the same encoder.

Usage

import numpy as np, soundfile as sf
from fujielab.asr.espnet_ext.espnet2.bin.asr_ctc_inference_cbs import Speech2TextCTC

s2t = Speech2TextCTC.from_pretrained(
    "fujie/espnet_asr_csj_pron_span_cbs_ctc_120300_hop132", streaming=True)

audio, fs = sf.read("utterance.wav")  # 16 kHz mono
chunk = int(16000 * 0.1)
for i in range(0, len(audio), chunk):
    c = audio[i:i+chunk]; is_final = len(c) < chunk
    if is_final: c = np.pad(c, (0, chunk - len(c)))
    r = s2t.streaming_decode(c, is_final=is_final)[0]
    print(" ".join(r.tokens))   # tokens carry the F/D markers

Training

CSJ (Corpus of Spontaneous Japanese), 30 epochs, ESPnet (espnet-ninjal fork); F/D labels derived from the CSJ transcripts.

Downloads last month: 2