| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - text-to-speech |
| - tts |
| - voice-cloning |
| - coreml |
| - apple-silicon |
| - chatterbox |
| library_name: coreml |
| base_model: ResembleAI/chatterbox |
| inference: false |
| --- |
| |
| # Chatterbox TTS — CoreML |
|
|
| CoreML port of [Chatterbox TTS](https://github.com/resemble-ai/chatterbox) by |
| Resemble AI. The full pipeline (T3 transformer → s3 speech tokenizer → s3gen |
| flow + flow estimator → mel-to-wav vocoder, plus the voice encoder and |
| CAMPPlus speaker encoder used for one-shot voice cloning) compiled to |
| `.mlpackage` for on-device inference on Apple Silicon Macs. |
|
|
| ## Intended use |
|
|
| Drop-in for Apple platform apps that want offline, on-device TTS with optional |
| voice cloning. Built specifically to back the narration feature in |
| [Screen Cut Pro](https://example.invalid) — but the files are the model only; |
| nothing here is app-specific. |
|
|
| ## Files |
|
|
| | File | Role | |
| | ----------------------------------- | ---------------------------------------------------------- | |
| | `t3_text_emb.mlpackage` | T3 text token embedding | |
| | `t3_text_pos_emb.mlpackage` | T3 text positional embedding | |
| | `t3_speech_emb.mlpackage` | T3 speech-token embedding | |
| | `t3_speech_pos_emb.mlpackage` | T3 speech-token positional embedding | |
| | `t3_cond_enc.mlpackage` | T3 conditioning encoder (speaker + emotion + prompt) | |
| | `t3_tfmr.mlpackage` | T3 Llama-style transformer (autoregressive backbone) | |
| | `t3_speech_head.mlpackage` | T3 speech-token output projection | |
| | `s3_tokenizer.mlpackage` | s3 speech tokenizer (mel → speech tokens) | |
| | `voice_encoder.mlpackage` | T3 speaker encoder (256-d embedding) | |
| | `campplus.mlpackage` | CAMPPlus speaker encoder for s3gen (192-d embedding) | |
| | `flow_encoder.mlpackage` | s3gen flow encoder | |
| | `flow_estimator.mlpackage` | s3gen flow estimator (diffusion) | |
| | `mel2wav.mlpackage` | Vocoder (mel → 24 kHz waveform) | |
| | `default_t3_speaker_emb.bin` | Bundled default voice — T3 speaker embedding | |
| | `default_t3_cond_prompt_tokens.bin` | Bundled default voice — T3 conditioning prompt tokens | |
| | `default_flow_prompt_token.bin` | Bundled default voice — s3gen prompt tokens | |
| | `default_flow_prompt_feat.bin` | Bundled default voice — s3gen prompt features | |
| | `default_flow_speaker_embedding.bin`| Bundled default voice — s3gen speaker embedding | |
|
|
| ## Conversion notes |
|
|
| Converted from the upstream PyTorch checkpoints with `coremltools`. A few |
| non-obvious patches were required to make the ONNX→CoreML path work: |
|
|
| - **s3 tokenizer RoPE** — the upstream uses `complex64` rotary embeddings, |
| which CoreML does not support. Replaced with split real `(cos, sin)` |
| tensors at trace time. |
| - **Fixed-length traces** — CoreML traces shapes statically; the flow encoder |
| is traced at 400 prompt tokens / 1024 mel frames. Inputs longer than that |
| must be chunked by the host application. |
| - **Mel features** — the upstream uses `librosa.filters.mel(slaney)` for |
| voice encoding and `kaldi.fbank(htk)` for CAMPPlus. The conversion script |
| bakes those windowing assumptions into the model where possible; the host |
| app reproduces them where not. See `Tools/compare_mels.py` in the source |
| app for verification. |
|
|
| The conversion script is in the source app's `Tools/` directory: |
| [`convert_chatterbox_to_coreml.py`](https://example.invalid). |
|
|
| ## License |
|
|
| MIT, inherited from upstream Chatterbox. See [`LICENSE`](./LICENSE) for the |
| full text and [`NOTICE`](./NOTICE) for attribution and a summary of |
| modifications. |
|
|
| ## Citation |
|
|
| If you use this in research, cite the upstream model: |
|
|
| ```bibtex |
| @misc{chatterbox2024, |
| title = {Chatterbox}, |
| author = {Resemble AI}, |
| year = {2024}, |
| url = {https://github.com/resemble-ai/chatterbox} |
| } |
| ``` |
|
|
