CSJ CTC-only Multitask ASR (recognition + auxiliary-information detection)

Streaming CTC model for Japanese that recognizes kana tokens and, for every token, predicts an auxiliary-information label:

N = normal
F = filler (フィラー)
D = repair / disfluency (言い直し)

Designed for use with fujielab-asr (>= 0.1.4).

Architecture

Encoder: Contextual Block Streaming Transformer (left context 12 / main block 3 / look-ahead 0 → block_size=15, hop_size=3, look_ahead=0), output_size 256, 12 blocks.
No decoder — base tokens recognized by CTC alone.
Auxiliary head: a linear layer on the encoder output predicting N/F/D per token. Training supervises per-token aux labels on the CTC forced-aligned frames; inference reads the aux head at each token's greedy-CTC emission frames.
Frontend hop_length 132 (8.25 ms frame shift @ 16 kHz); word-level kana tokens.

Results (CSJ eval1/2/3, 10-best averaged model, greedy CTC)

set	CER	Filler F1	Repair F1
eval1	7.87%	91.7	45.9
eval2	5.81%	91.4	43.7
eval3	6.30%	88.6	38.7
all	6.71%	90.9	43.7

Usage

import numpy as np, soundfile as sf
from fujielab.asr.espnet_ext.espnet2.bin.asr_multitask_ctc_inference_cbs import (
    Speech2TextMultitaskCTC,
)

s2t = Speech2TextMultitaskCTC.from_pretrained(
    "fujie/espnet_asr_csj_pron_aux_cbs_ctc_120300_hop132", streaming=True
)

audio, fs = sf.read("utterance.wav")  # 16 kHz mono
chunk = int(16000 * 0.1)              # 100 ms
for i in range(0, len(audio), chunk):
    c = audio[i:i+chunk]
    is_final = len(c) < chunk
    if is_final:
        c = np.pad(c, (0, chunk - len(c)))
    r = s2t.streaming_decode(c, is_final=is_final)[0]
    print(" ".join(f"{t}[{a}]" if a != "N" else t
                    for t, a in zip(r.tokens, r.aux_labels)))

See examples/run_streaming_asr_multitask.py in fujielab-asr.

Training

Trained on CSJ (Corpus of Spontaneous Japanese), train_nodup (~450k utterances), 30 epochs, with ESPnet (espnet-ninjal fork). The auxiliary labels are derived from the CSJ transcripts (filler / disfluency tags).

Downloads last month: 6