File size: 4,314 Bytes
be223ae | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
license: mit
language:
- en
tags:
- text-to-speech
- tts
- voice-cloning
- coreml
- apple-silicon
- chatterbox
library_name: coreml
base_model: ResembleAI/chatterbox
inference: false
---
# Chatterbox TTS — CoreML
CoreML port of [Chatterbox TTS](https://github.com/resemble-ai/chatterbox) by
Resemble AI. The full pipeline (T3 transformer → s3 speech tokenizer → s3gen
flow + flow estimator → mel-to-wav vocoder, plus the voice encoder and
CAMPPlus speaker encoder used for one-shot voice cloning) compiled to
`.mlpackage` for on-device inference on Apple Silicon Macs.
## Intended use
Drop-in for Apple platform apps that want offline, on-device TTS with optional
voice cloning. Built specifically to back the narration feature in
[Screen Cut Pro](https://example.invalid) — but the files are the model only;
nothing here is app-specific.
## Files
| File | Role |
| ----------------------------------- | ---------------------------------------------------------- |
| `t3_text_emb.mlpackage` | T3 text token embedding |
| `t3_text_pos_emb.mlpackage` | T3 text positional embedding |
| `t3_speech_emb.mlpackage` | T3 speech-token embedding |
| `t3_speech_pos_emb.mlpackage` | T3 speech-token positional embedding |
| `t3_cond_enc.mlpackage` | T3 conditioning encoder (speaker + emotion + prompt) |
| `t3_tfmr.mlpackage` | T3 Llama-style transformer (autoregressive backbone) |
| `t3_speech_head.mlpackage` | T3 speech-token output projection |
| `s3_tokenizer.mlpackage` | s3 speech tokenizer (mel → speech tokens) |
| `voice_encoder.mlpackage` | T3 speaker encoder (256-d embedding) |
| `campplus.mlpackage` | CAMPPlus speaker encoder for s3gen (192-d embedding) |
| `flow_encoder.mlpackage` | s3gen flow encoder |
| `flow_estimator.mlpackage` | s3gen flow estimator (diffusion) |
| `mel2wav.mlpackage` | Vocoder (mel → 24 kHz waveform) |
| `default_t3_speaker_emb.bin` | Bundled default voice — T3 speaker embedding |
| `default_t3_cond_prompt_tokens.bin` | Bundled default voice — T3 conditioning prompt tokens |
| `default_flow_prompt_token.bin` | Bundled default voice — s3gen prompt tokens |
| `default_flow_prompt_feat.bin` | Bundled default voice — s3gen prompt features |
| `default_flow_speaker_embedding.bin`| Bundled default voice — s3gen speaker embedding |
## Conversion notes
Converted from the upstream PyTorch checkpoints with `coremltools`. A few
non-obvious patches were required to make the ONNX→CoreML path work:
- **s3 tokenizer RoPE** — the upstream uses `complex64` rotary embeddings,
which CoreML does not support. Replaced with split real `(cos, sin)`
tensors at trace time.
- **Fixed-length traces** — CoreML traces shapes statically; the flow encoder
is traced at 400 prompt tokens / 1024 mel frames. Inputs longer than that
must be chunked by the host application.
- **Mel features** — the upstream uses `librosa.filters.mel(slaney)` for
voice encoding and `kaldi.fbank(htk)` for CAMPPlus. The conversion script
bakes those windowing assumptions into the model where possible; the host
app reproduces them where not. See `Tools/compare_mels.py` in the source
app for verification.
The conversion script is in the source app's `Tools/` directory:
[`convert_chatterbox_to_coreml.py`](https://example.invalid).
## License
MIT, inherited from upstream Chatterbox. See [`LICENSE`](./LICENSE) for the
full text and [`NOTICE`](./NOTICE) for attribution and a summary of
modifications.
## Citation
If you use this in research, cite the upstream model:
```bibtex
@misc{chatterbox2024,
title = {Chatterbox},
author = {Resemble AI},
year = {2024},
url = {https://github.com/resemble-ai/chatterbox}
}
```
|