--- license: cc-by-sa-4.0 base_model: FunAudioLLM/CosyVoice2-0.5B tags: - nanites - tts - text-to-speech - cosyvoice2 - anonymized-voice - british-isles - scottish - irish - welsh - dialect language: - en pipeline_tag: text-to-speech --- # nanites-isles-speech-{hamish,maisie,seamus,tegan,rhys} Anonymized, dialect+gender-pooled British Isles English text-to-speech voices — Scottish, Irish, and Welsh English as crowdsourced in the **OpenSLR 83** dataset (Google, 2018–19). Built on [CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) (Apache-2.0, from FunAudioLLM), fine-tuned separately per pool. ## Personas Pooled voices ship under dialect-appropriate codenames so each identity is a *persona*, never a single OpenSLR speaker: | Persona | Dialect/gender | Pool | WER (medium.en) | Anon max cos | |---|---|---|---|---| | **Maisie** | Scottish female | 6 speakers / 54.4 min | 26%* | 0.911 | | **Hamish** | Scottish male | 6 speakers / 53.4 min | 22%* | 0.782 | | **Seamus** | Irish male | 3 speakers / 22.7 min (thin) | 4.2% | 0.895 | | **Tegan** | Welsh female | 6 speakers / 60.1 min | 8.9% | 0.871 | | **Rhys** | Welsh male | 6 speakers / 59.8 min | 0.0% | 0.883 | \* Scots lines are deliberately written in Scots orthography (e.g. "dinna fash", "ged your tea"); whisper normalizes them to standard spelling, so WER is inflated by orthography, not speech errors. The Irish/Welsh lines use standard English spelling and their WERs are reflective. In code/config the voices keep the functional names `isles-speech-scottish-female` etc. (engine + dialect + gender are metadata); the codenames are the display/persona names. **Hugging Face repos are persona-named** (a persona gets its own repo; a future voice gets a new codename, not a gender suffix): - `Nanite-Labs/nanites-isles-speech-hamish` (Scottish male) - `Nanite-Labs/nanites-isles-speech-maisie` (Scottish female) - `Nanite-Labs/nanites-isles-speech-seamus` (Irish male) - `Nanite-Labs/nanites-isles-speech-tegan` (Welsh female) - `Nanite-Labs/nanites-isles-speech-rhys` (Welsh male) ## Why this approach - **Engine.** Same rationale as the sister `smoky` suite: modern line is **CosyVoice 2.0** (FunAudioLLM), Apache-2.0. **XTTS is retired** in this project (dormant, CPML license, poor clustering/cloning control). - **Identity.** A *regional anon voice* must not replicate any single OpenSLR speaker. CosyVoice2 SFT collapses each dialect+gender pool into **one pooled speaker ID** (`<|{pool}_pool|>`), so no single speaker is recoverable at synthesis time. - **Anonymization.** Generation feeds the pooled campplus embedding (average of the pool's members) — no reference audio from any individual. Max cosine of the pooled embedding to the closest member is reported per voice above (0.78–0.91; smoky reference ~0.89–0.91). **Seamus's pool is thin** (3 speakers — the smallest OpenSLR set; the Irish female pool does not exist in OpenSLR 83, so Irish female is not offered): its anonymization margin and training-data size are inherently smaller than the 6-speaker pools. ## Stable persona (zero-shot anchoring) SFT mode conditions identity only through the pooled embedding, and the base speech flow can leak timbre from the token sequence — some short/locked-phrase sentences **deterministically** land in a different timbre region (diagnosed on Hamish: one sentence regen 3× identical but cosine 0.58 from his anchor). Fix, adopted for all five voices: **zero-shot anchoring.** Each voice dir carries `anchor.wav` + `anchor.txt` (a pooled-blend clip chosen by the listener, so the anchor is itself anonymized); `cosyvoice_infer` clones that clip's timbre for every new sentence (cosine 0.83–0.94 vs the anchor across novel lines). Inference is therefore persona-consistent per codename. ## Data & license - **OpenSLR 83** — "Crowdsourced high-quality UK and Ireland English Dialect speech data set" (Google, 2018–19), **CC BY-SA 4.0**. Citation: Demirsahin et al., "Open-source Multi-speaker Corpora of the English Accents in the British Isles", LREC 2020. - **CC BY-SA carries over to these voices**: derivatives (incl. synthesized redistribution) inherit share-alike. This differs from smoky (public-domain Hall corpus). - OpenSLR 83 offers 6 dialects × male/female; this release covers Scottish (f+m), Irish (male), Welsh (f+m). The Irish female set is not offered because OpenSLR 83 only ships `irish_english_male.zip`. - Audio: 48 kHz mono studio recordings → pooled 24 kHz clean sets with light spectral-gate denoise, energy-VAD trim, SNR ≥ 8 dB filter, per-pool caps (6 speakers / ~60 min / 10 min per speaker). ## Training - **Base:** `FunAudioLLM/CosyVoice2-0.5B` (llm + flow + hift). - **Data:** pooled chunk sets → kaldi-style dirs (10% cv) → campplus embeddings + speech tokens → parquet → `cosyvoice/bin/train.py` (`--model llm`, 1×GPU, bf16 AMP, constant-lr 1e-5, 60 epochs). One speaker ID per pool. - **Guarding (8 GB GPU):** bitsandbytes AdamW8bit; non-LLM params frozen; bf16; gradient checkpointing; `KEEP_CKPT=12` in-run pruning of checkpoints (multi-speaker SFT writes ~1 GB/epoch → ~60 GB if unpruned). - **Outputs:** `llm_avg.pt` per voice (**best-8 by val loss** via `average_model.py --val_best`); final voice dirs assembled with base flow/hift + baked `spk2info.pt` + anchor + samples. ## Evaluation - WER: whisper `medium.en` on novel pool dialect lines (see table; target <12%; Scots lines orthography-inflated). Tools: `experts/isles/eval_wer.py`, `eval_texts.py`. - Anonymization: `check_anon.py` — pooled embedding max cosine to a member, per voice (see table). - Persona consistency: anchor-mode synthesis lands at cosine 0.83–0.94 vs the anchor across novel lines (vs 0.58 drift pre-fix). ## Report to User - [x] Whisper WER < 12% on held-out lines (per voice; see table) - [x] Anonymization check (max cos 0.78–0.91) - [x] Persona-consistent anchored synthesis (0.83–0.94 vs anchor) - [x] All five voices signed off by listening; training data pruned - [ ] Publish zips via `publish_cosyvoice.py --voice ...` (or all)