nanites-smoky-speech-otis / nanites-smoky-speech-opal

Anonymized, gender-pooled Smoky Mountain English text-to-speech voices — the 1930s regional dialect of the Great Smoky Mountains (NC + TN) as recorded by Joseph Sargent Hall in 1939. Built on Fun-CosyVoice3-0.5B-2512 (Apache-2.0, from FunAudioLLM), fine-tuned separately per gender. Second generation of the Smoky voices: new randomly-sampled speaker pools on the CosyVoice 3 engine (first generation: Earl + Ethel on CosyVoice2).

Personas

The two pooled voices ship under Appalachian codenames so the identity is a persona, never a single Hall speaker:

• Otis (male) — 15 speakers, 265 clips, 40.0 min. Best-8 epochs [10, 8, 9, 12, 13, 15, 11, 6] by val loss. Samples: sample/otis_0.wav, sample/otis_1.wav, sample/otis_2.wav (chatbot-written lines, same three lines as Opal for comparison).
• Opal (female) — 15 speakers, 160 clips, 24.8 min. Best-8 epochs [8, 14, 15, 13, 9, 12, 11, 16] by val loss. Samples: sample/opal_0.wav, sample/opal_1.wav, sample/opal_2.wav.

In code/config the voices keep functional names smoky-speech-male_v2 / smoky-speech-female_v2 with speaker IDs <|male_pool_v2|> / <|female_pool_v2|>; "Otis" and "Opal" are the display/persona names. Hugging Face repos are persona-named:

• Nanite-Labs/nanites-smoky-speech-otis
• Nanite-Labs/nanites-smoky-speech-opal

Pools (random subsets, seed 20260925)

Male v2 is fully disjoint from Earl (0 overlap). Female v2 reuses 10 of Ethel's speakers (only 6 unused female codes existed in the archive) plus 5 fresh ones (GladysHoyle, sr011am, sr038am, sr049bm, sr084am):

• male: sr012bm, sr042am, sr042bm, sr048am, sr050bm, sr057bm, sr058am, sr062bm, sr064bm, sr067bm, sr074am, sr076am, sr076bm, sr102bm, sr161bm
• female: GladysHoyle, sr011am, sr011bm, sr033bm, sr038am, sr049am, sr049bm, sr060am, sr066bm, sr068am, sr075am, sr075bm, sr084am, sr086am, sr086bm

Pipeline per clip (same as v1): 16 kHz → 24 kHz resample, spectral-gate denoise, energy-VAD trim, SNR filter (≥ 8 dB).

Training

• Base: FunAudioLLM/Fun-CosyVoice3-0.5B-2512 (llm.pt + flow.pt + hift.pt)
• Data: pooled chunk sets → kaldi-style dirs (wav.scp/text/utt2spk + per-utterance instruct: You are a helpful assistant.<|endofprompt|>, required by the v3 LLM) → campplus embeddings + v3 speech tokens → parquet → cosyvoice/bin/train.py (--model llm, 1×GPU, bf16, constant-lr 1e-5, use_spk_embedding: True). Single speaker ID per gender.
• 8 GB recipe: bitsandbytes AdamW8bit; whole model bf16 (no GradScaler); gradient checkpointing (COSYVOICE_GC=1); DDP wrap skipped at world_size 1; single-proc join guard. Patches live in experts/smoky/train_sft.py (repo clone stays pristine).
• Outputs: llm_avg.pt per gender (best-8 by val loss); voice dirs assembled from the v3 base + averaged LLM + baked spk2info.pt.
• Inference note (v3): synthesis text must carry the instruct prefix — You are a helpful assistant.<|endofprompt|> is prepended automatically by experts/smoky/cosyvoice_infer.py for v3 voice dirs. SFT mode only (inference_sft, pooled embedding, no reference clip).

python experts/smoky/cosyvoice_infer.py \
    --model_dir <voice-dir> \
    --text "What was sharecropping like?" \
    --sft_spk_id "<|male_pool_v2|>" \
    --out out.wav

Evaluation

• WER (whisper medium.en, 3 novel chatbot-written dialect lines, 132 words): Otis 6.1%, Opal 7.6%. Target < 12%. (Residual errors are dialect spellings: crick→creek, dawnin→dawning.)
• Anonymization: pooled campplus embedding max cosine to the closest pool-member mean: 0.924 (male) / 0.927 (female) — a blended mean identity matching no single speaker (v1: 0.888/0.909).
• Sample lines were written by the collection's own chat model (nanites-smoky-1b-chat) so text and speech share the regional register.

Provenance

Audio: public-domain 1939 Hall field recordings (USC archive). Transcripts (Montgomery & Reed 2017, copyrighted) used locally only — never redistributed; training artifacts are derived phonetic tokens, not source text. Pooling is structural anonymization: one speaker ID per gender, no reference audio at inference.