infinia-ai/magpie-tts-13lang-357m

Magpie-TTS 13-Language (357M) — with Swahili

📄 Paper: Cross-Lingual Grafting: Extending Frozen Codec Speech Language Models to Unseen Low-Resource Languages — PDF.

A single 13-language text-to-speech checkpoint: the twelve languages of NVIDIA's magpie_tts_multilingual_357m (Arabic, German, English, Spanish, French, Hindi, Italian, Japanese, Korean, Portuguese, Vietnamese, Chinese) plus Swahili (Kiswahili) — added by a community team without regressing any of the original twelve.

Swahili was grafted onto the frozen base model with a dedicated byte-level tokenizer, a warm-started input-embedding surgery, and multilingual rehearsal. On held-out Swahili the model reaches 8.8% character error rate (median 0.0%, MMS-Swahili ASR), and the twelve base languages show no measurable regression versus the untouched base. Full method: see the accompanying paper "Cross-Lingual Grafting: Extending Frozen Codec Speech Language Models to Unseen Low-Resource Languages" (New Emerging Technologies, a subsidiary of Infinia Technologies).

⚠️ Community model. Swahili support was added by the community and is not provided or endorsed by NVIDIA. The twelve base languages follow the base model.

Highlights

Base	Magpie-TTS-Multilingual 357M (Koel-TTS family)
Audio codec	Low Frame-rate Speech Codec (22.05 kHz, 1.89 kbps, 21.5 fps)
Languages	13 (12 base + Swahili)
Swahili quality	8.8% mean / 0.0% median CER (MMS-sw)
Base regression	~0 (mean recognizer CER identical to base)
Voices	5 baked speakers (2 female, 3 male), selected by index
Trained on	1× 128 GB unified-memory GPU

Usage (NVIDIA NeMo)

from nemo.collections.tts.models import MagpieTTSModel
import soundfile as sf, numpy as np, torch, random

m = MagpieTTSModel.from_pretrained("infinia-ai/magpie-tts-13lang-357m").eval().cuda()

# Deterministic inference: pin the seed so identical inputs give identical audio
def seed(s=1234):
    random.seed(s); np.random.seed(s); torch.manual_seed(s); torch.cuda.manual_seed_all(s)

seed(1234)
audio, alen = m.do_tts(
    "Habari ya asubuhi. Karibu kwenye jaribio la sauti ya Kiswahili.",
    language="sw",            # native Swahili code; also en, de, es, fr, it, vi,
                              # zh, hi, ja, pt-BR, ko, ar-MSA, ar-SA, ar-AE
    speaker_index=0,          # 0 Aria(F) 1 Jason(M) 2 John(M) 3 Leo(M) 4 Sofia(F)
    apply_TN=False, use_cfg=True,
)
w = np.squeeze(audio.detach().float().cpu().numpy())
sf.write("out.wav", (w[0] if w.ndim > 1 else w).astype(np.float32), 22050)

Determinism. The released tokenizers use phoneme_probability=1.0 (deterministic phonemization) and the sampler defaults (temperature 0.7, top-k 80, CFG 2.5) match the base model. With a fixed RNG seed, identical (text, language, voice, CFG, seed) inputs produce byte-identical audio; change the seed for a different rendering. For the occasional degenerate silent generation, resample at a new seed (retry-on-silence).

Language codes

en, de, es, fr, it, vi, zh, hi, ja, pt-BR, ko, ar-MSA, ar-SA, ar-AE, sw.

Training data

• Swahili: Bateesa/kiswahili-tts-dataset (CC-BY) + FLEURS sw_ke (CC-BY), ~27.8 h, resampled to 22.05 kHz.
• Rehearsal (12 base languages): FLEURS, ~500 utterances/language, each routed to its native tokenizer.

Please cite FLEURS (Conneau et al., arXiv:2205.12446) and the Koel-TTS (arXiv:2502.05236) and Low Frame-rate Speech Codec (arXiv:2409.12117) papers.

Intended use & limitations

Research and product speech synthesis for the 13 supported languages. Out of scope: impersonation of real individuals, deceptive or harmful synthetic media. The model synthesizes a fixed set of 5 baked voices; it does not support zero-shot voice cloning (the base model's context encoder is not included). Swahili data skews toward read/literary speech and one dominant speaker; FLEURS adds multi-speaker breadth at upsampled 16 kHz. 357M parameters bound absolute quality.

Responsible use

Generated audio carries the base model's watermark and synthetic-speech disclosure. Do not use to deceive. Disclose that audio is AI-generated.

License & attribution

Released under the NVIDIA Open Model License (see LICENSE and NOTICE). Derived from nvidia/magpie_tts_multilingual_357m. "Licensed by NVIDIA Corporation under the NVIDIA Open Model License." Training data: FLEURS (CC-BY), Bateesa Kiswahili TTS (CC-BY).