`fujie/espnet_asr_csj_cejc_pron_aux_cbs_transducer_120300_hop132`

Streaming Japanese ASR that also marks fillers and repairs, trained on CSJ + CEJC by Shinya Fujie (Fujie Lab, Chiba Institute of Technology).

This is the multitask Transducer (separate N/F/D head) variant. The transcript is plain kana. The auxiliary information is a parallel label sequence (one of N / F / D per recognized token), predicted by a separate head, so the text itself stays free of markup.

Two sibling models encode the same information differently — see the comparison below:

fujie/espnet_asr_csj_cejc_pron_aux_cbs_transducer_120300_hop132 — separate N/F/D head
fujie/espnet_asr_csj_cejc_pron_comp_cbs_transducer_120300_hop132 — composite tokens
fujie/espnet_asr_csj_cejc_pron_span_cbs_transducer_120300_hop132 — span tags

Usage

Runs with current ESPnet via fujielab-asr — no old-commit checkout needed.

pip install "fujielab-asr>=0.2.0"

import numpy as np, soundfile as sf
from fujielab.asr.espnet_ext.espnet2.bin.asr_multitask_transducer_inference_cbs import (
    Speech2TextMultitaskTransducer,
)

s2t = Speech2TextMultitaskTransducer.from_pretrained(
    "fujie/espnet_asr_csj_cejc_pron_aux_cbs_transducer_120300_hop132",
    beam_size=20, beam_search_config=dict(search_type="maes"),
)

audio, fs = sf.read("utterance.wav")   # 16 kHz mono
chunk = int(16000 * 0.1)               # 100 ms
n, result = len(audio), None
for i in range(0, n, chunk):
    c = audio[i:i + chunk]
    is_final = i + chunk >= n          # flag the last chunk even when it is full
    if len(c) < chunk:
        c = np.pad(c, (0, chunk - len(c)))
    r = s2t.streaming_decode(c, is_final=is_final)
    if r:
        result = r[0]

# tokens and aux_labels are aligned 1:1
print(" ".join(f"{t}[{a}]" if a != "N" else t
               for t, a in zip(result.tokens, result.aux_labels)))

Only search_type="maes" is exercised by these models.

Streaming vs. offline auxiliary labels

The auxiliary head sits on the joint network's pre-output hidden state, so a label only exists at a specific (frame, predictor-state) node of the Transducer lattice. The offline research recipe picks that node by re-encoding the whole utterance and forced-aligning the final hypothesis — a second pass over complete audio, which a streaming recognizer cannot perform.

fujielab-asr instead reads the head at the node the beam search actually emitted the token at. That needs no second pass, so auxiliary labels are available on every 100 ms chunk rather than only at end of utterance.

Measured agreement between the two protocols on 297 held-out CEJC utterances (4,619 tokens), holding the hypothesis and the encoder output fixed so that only the choice of node varies:

	agreement
all tokens	99.50 %
tokens where either protocol says `F` or `D`	90.17 %

Emission frames coincide exactly for 54 % of tokens and land within one frame (33 ms) for 92 %. So the streaming labels track the reported offline numbers closely, but they are not bit-identical to them — worth knowing if you are reproducing the paper figures.

Evaluation

17 evaluation sets (CEJC 14 + CSJ 3), 23,500 utterances. Decoded with valid.cer_transducer.ave_10best.pth, beam 20.

CER is computed with all auxiliary information stripped from both reference and hypothesis, so the three encodings are measured against the same target. Scoring the markup itself would penalise comp and span for text the aux-head model never has to emit, which is what made an earlier comparison look like a 2–3 point gap when the real gap is under half a point.

	CER %	F: P / R / F1	D: P / R / F1
this model	16.01	82.6 / 81.1 / 81.9	59.5 / 35.4 / 44.4

By corpus: CEJC 22.60 % / CSJ 5.36 % CER.

F = filler (フィラー), D = repair / disfluent restart (言い直し). Detection counts as correct when the predicted span overlaps the reference span of the same type within ±1 token.

The three encodings compared

All three models share the encoder geometry, training data and schedule, and differ only in how the auxiliary information is encoded:

encoding	CER %	F F1	D F1
`aux` (separate head)	16.01	81.9	44.4
`comp` (`ア+F`)	16.25	82.0	45.5
`span` (`<F> … </F>`)	16.41	81.5	43.9

A paired bootstrap over utterances calls every pairwise CER gap significant, but the comp vs span gap flips sign between the 14-epoch and 50-epoch checkpoints (+0.14 → −0.17), so run-to-run variance exceeds it: treat those two as indistinguishable. aux is ahead of both at either training length, though by only 0.23–0.40 points. comp has the best D detection at both lengths.

D recall is 35–38 % across all three — the weak point of every model here. Predictions are reasonably precise (55–60 %) but miss most repairs.

Training

Encoder: Contextual Block Conformer, 12 blocks, d=256, 4 heads, macaron + conv module (kernel 15). Left context 12 / main block 3 / look-ahead 0 — i.e. no look-ahead, so the encoder is causal at block granularity. block_size 15, hop_size 3.
Decoder: RNN-Transducer, single-layer LSTM predictor (hidden 512), joint space 640.
Frontend: 80-dim log-mel, hop_length 132 samples (8.25 ms) at 16 kHz, ×4 subsampling → one encoder frame per 33 ms.
Data: CSJ + CEJC, kana (pronunciation) tokens. Fillers are identified with the CSJ-aligned criterion, which also treats short vowel fillers (えー / まあ / あー / ん) as F — a harder detection target than the UniDic-based criterion, and one that costs about 8 points of F F1 while leaving CER unchanged.
Schedule: 50 epochs × 1400 steps, Adam lr 0.0035, warmup 2500, effective batch 32 M bins (16 M × 2 × H100).

14 epochs was substantially undertrained for this setup: extending to 50 moved CER by 2.4–2.9 points and D F1 by 7–12 points, far more than the choice of auxiliary encoding does.

Note on positional encoding and long audio

The encoder adds an absolute sinusoidal positional encoding indexed from the start of the stream, and its self-attention is the plain (not relative) variant. Training utterances here run to 19.9 s at most (498 encoder frames), so feeding a long continuous stream pushes the position index far outside the trained range. Segment the audio (e.g. by VAD) and let the recognizer reset per segment rather than streaming for minutes on end.

Citing ESPnet

@inproceedings{watanabe2018espnet,
  author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi
          and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann
          and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala
          and Tsubasa Ochiai},
  title={{ESPnet}: End-to-End Speech Processing Toolkit},
  year={2018},
  booktitle={Proceedings of Interspeech},
  pages={2207--2211},
  doi={10.21437/Interspeech.2018-1456}
}

Downloads last month: 5