chatterbox-cymraeg — Welsh TTS (Chatterbox Multilingual T3, full fine-tune)

Welsh (Cymraeg) text-to-speech with voice cloning, from a full fine-tune of Resemble AI's Chatterbox Multilingual T3 on 100 hours of Common Voice Welsh.

Inference must pass language_id="fi". This is not a mistake — see The [fi] language tag below. Passing "cy" raises ValueError.

Contents

Path	What
step13971/t3_mtl23ls_v2.safetensors	fine-tuned T3, the only trained component
step13971/s3gen.pt, step13971/ve.pt	S3Gen flow decoder and voice encoder, unchanged from base
step13971/grapheme_mtl_merged_expanded_v1.json, Cangjie5_TC.json	tokenizer, unchanged from base
step13971/export_manifest.json	provenance: source checkpoint, global step, what was trained
eval/	full objective evaluation, per-set summaries and per-row scores
gates/	the voicing/duration gate tables from training and the head-to-head

Usage

from chatterbox.mtl_tts import ChatterboxMultilingualTTS
import torchaudio as ta

model = ChatterboxMultilingualTTS.from_local("step13971", device="cuda")
wav = model.generate(
    "Mae'r gwersi yn canolbwyntio ar ysgrifennu i berswadio.",
    language_id="fi",                      # REQUIRED — see below
    audio_prompt_path="reference_speaker.wav",
    temperature=0.8, cfg_weight=0.3, exaggeration=0.5, repetition_penalty=1.5,
)
ta.save("out.wav", wav, model.sr)

The decoding settings above are the ones every number on this card was measured with.

The [fi] language tag

Chatterbox Multilingual supports 23 languages; Welsh is not one of them, and its grapheme tokenizer has no [cy] token (vocabulary of 2,454). The nine [PLACEHOLDER55..63] slots that exist are unreachable, because MTLTokenizer.encode lowercases the tag it constructs, so [PLACEHOLDER55] can never be produced.

So the Finnish slot was reused as a donor tag and re-specialised by the fine-tune. Finnish was picked because it is the slot with published evidence of taking a full new-language fine-tune (Finnish-NLP/Chatterbox-Finnish) and because it shares Welsh's transparent orthography and non-final stress. The model saw only [fi] during training, so the tag now means Welsh to it.

Welsh graphemes are otherwise fully covered: the tokenizer NFKD-normalises before encoding, so ŵ and ŷ decompose into base letter plus combining circumflex and every Welsh grapheme round-trips without [UNK].

Training

Base	ResembleAI/chatterbox, t3_mtl23ls_v2.safetensors
Trained	T3 only (0.5B Llama token predictor), all parameters
Frozen	S3Gen flow decoder, voice encoder, tokenizer
Data	techiaith/commonvoice_16_1_cy @ 910ee33ed03c3f2edf554df6b15f08f4deac8f3c (CC0-1.0)
Corpus	74,505 clips, 100.4 h, 1,475 speakers, 24 kHz mono
Splits	speaker-disjoint and sentence-disjoint; validation 427 rows / 15 speakers held out
Recipe	AdamW, LR 2e-5 cosine, 200 warmup, bf16, 16 utterances/step
Length	3 epochs = 13,971 steps, final train loss 1.922
Hardware	2× RTX 3090, DDP via accelerate, ~1.2 s/step, ~4.9 h total
Toolkit	99eren99/chatterbox-multilingual-finetuning

Freezing S3Gen is the design decision that matters. It is the component that carries voicing, so keeping it frozen puts the acoustic path out of the optimiser's reach; only the token predictor learns Welsh.

Evaluation

Three frozen evaluation sets, 97 prompts. Identical prompt files, conditioning references (never a prompt's own recording) and per-prompt seeds across every model below. Objective ASR is techiaith/whisper-large-ft-cy-en @ 014eacd47a907794bf093d033488c5edff9cdf58, greedy, language=cy. WER/CER are NFC + casefold + punctuation-stripped, via jiwer.

Voicing (median pitch-band autocorrelation over voiced frames) is reported first and gates the rest: an ASR model will hallucinate fluent Welsh over degenerate audio, so a WER is only meaningful once the audio is known not to be mush. All three sets pass (≥ 0.50).

Evaluation set	n	voicing	this model WER	Chatterbox base WER	mean CER
cv16_stage1_test (unseen speakers + text)	40	0.534	0.263	0.663	0.088
bangor_code_switch (Welsh/English)	27	0.556	0.378	0.714	0.129
fleurs_cy_gb_test (external read speech)	30	0.595	0.295	0.703	0.122

Against the untouched Chatterbox base under identical decoding, that is a 60% / 47% / 58% relative reduction in mean WER.

Duration and truncation

set	mean generated/human	clips > 30 s	over-runs	rows < 0.5× human
cv16_stage1_test	0.74	0	0	1
bangor_code_switch	1.16	0	0	0
fleurs_cy_gb_test	0.66	0	0	7

Caveats

• It speaks faster than the reference humans — about 13% shorter than the human recordings on held-out Common Voice sentences (0.74× on cv16_stage1_test, against 0.845× for the untrained base). The fine-tune compressed delivery toward Common Voice's tighter clip style.
• Long, multi-sentence prompts can get clipped. Of 97 prompts, 8 came out under half the human duration; by WER, 4 are genuinely cut off (the rest are complete but brisk), and all the clipped ones are long FLEURS prompts. Note the untrained base clips 6 of the same 30 FLEURS prompts, so this is a property of the model family and decoding settings, not something the Welsh fine-tune introduced. Split long text into sentences.
• Voicing is level with base, not better: 0.569 against 0.564 on the six-prompt gate set. The fine-tune bought pronunciation, not audio quality — which is the expected result of freezing S3Gen, and the point of doing so.
• WER is one signal. The Welsh ASR has its own dialect and orthographic biases and was itself fine-tuned on Welsh corpora, so it is gentler on Welsh-shaped errors than a listener and harsher on legitimate dialect variation. Listen before trusting.
• Trained on read speech from Common Voice; not validated for expressive, conversational or long-form synthesis.
• Common Voice contributor deletion requests made after the 16.1 snapshot cannot be reflected in a frozen corpus.

Licences

• Weights: MIT, inherited from ResembleAI/chatterbox.
• Training data: CC0-1.0 (techiaith/commonvoice_16_1_cy).
• Generated audio carries Resemble AI's Perth implicit watermark, as base Chatterbox does.

Citation

Base model: Chatterbox, Resemble AI. Data: Common Voice Welsh 16.1, repacked by Techiaith, Bangor University.