README.md

---
license: other
license_name: tontaube-community-model-license-1.0
license_link: https://huggingface.co/TontaubeAI/TontaubeV1/blob/main/LICENSE
pipeline_tag: text-to-speech
base_model:
  - Qwen/Qwen3-1.7B
  - Qwen/Qwen3-0.6B
language:
  - en
  - de
  - es
  - fr
  - it
  - nl
  - pt
tags:
  - text-to-speech
  - voice-cloning
  - multilingual
  - audio
  - vllm
inference: false
---

<p align="center">
  <img src="assets/tontaube-logo.png" alt="Tontaube" width="180">
</p>

# TontaubeV1

TontaubeV1 is a multilingual text-to-speech model designed for expressive
voice cloning, long-form generation, and low-latency streaming. English and
German are its primary languages; Spanish, French, Italian, Dutch, and
Portuguese are also supported.

The release contains four causal codebook predictors. CB0 generates semantic
audio and duration from text, while progressively smaller CB1--CB3 models add
acoustic detail. Spoken text uses character-level tokenization, with compact
BPE-encoded language and speaking-style controls. The public styles are
`audiobook`, `conversational`, and `agentic`.

[Technical report](https://arxiv.org/abs/2609.08703) ·
[Try TontaubeV1 online](https://tontaube.ai/playground) ·
[Inference repository](https://github.com/craitech/tontaube)

## Quick start

These checkpoints require the Tontaube vLLM model adapter and input layout;
they are not drop-in `AutoModelForCausalLM` checkpoints. Use the matching
release of the inference repository:

```bash
git clone --branch v1.0.0 --depth 1 \
  https://github.com/craitech/tontaube.git
cd tontaube
uv run --python 3.12 --frozen tontaube serve
```

Once the API is ready, generate a WAV with the bundled English voice from a
second terminal:

```bash
curl --silent --show-error --fail http://127.0.0.1:8080/predict \
  -H 'content-type: application/json' \
  -d '{"text":"Welcome to Tontaube.","language":"english","tag":"conversational","format":"wav"}' \
| python3 -c 'import base64,json,sys; sys.stdout.buffer.write(base64.b64decode(json.load(sys.stdin)["audio_b64"]))' \
> welcome.wav
```

Clone a voice by sending about 5--60 seconds of clean, single-speaker speech
with the request:

```bash
VOICE_FILE=/path/to/reference.wav
python3 -c 'import base64,json,sys; print(json.dumps({"text":"This voice was cloned from a local reference.","language":"english","tag":"conversational","voice_audio_b64":base64.b64encode(open(sys.argv[1],"rb").read()).decode(),"format":"wav"}))' "$VOICE_FILE" \
| curl --silent --show-error --fail http://127.0.0.1:8080/predict \
  -H 'content-type: application/json' \
  --data-binary @- \
| python3 -c 'import base64,json,sys; sys.stdout.buffer.write(base64.b64decode(json.load(sys.stdin)["audio_b64"]))' \
> cloned.wav
```

The server downloads this model and its pinned runtime dependencies into local
caches on first use. Follow the inference repository's installation
instructions for system requirements, the browser interface, and Docker. The
optional English verbalizer is available separately from
`TontaubeAI/TontaubeV1-Verbalizer` and is downloaded only when enabled.

The default semantic sampling temperature is `0.8`. A value around `0.6`
generally favors text correctness, while values approaching `1.0` can produce
more varied and expressive delivery.

## Capabilities

- Bundled synthetic voices and zero-shot cloning from up to 60 seconds of
  reference audio
- `audiobook`, `conversational`, and `agentic` speaking styles
- Bounded-context long-form generation
- WAV generation and low-latency MP3 or Opus streaming
- Optional English text verbalizer, distributed separately

## Examples

These MP3 samples were normalized to a consistent speech listening level.

### Audiobook

**English 1**

<audio controls preload="none" src="https://huggingface.co/TontaubeAI/TontaubeV1/resolve/main/assets/showcase/english-audiobook-01.mp3"></audio>

[Download MP3](assets/showcase/english-audiobook-01.mp3)

**English 2**

<audio controls preload="none" src="https://huggingface.co/TontaubeAI/TontaubeV1/resolve/main/assets/showcase/english-audiobook-02.mp3"></audio>

[Download MP3](assets/showcase/english-audiobook-02.mp3)

**English 3**

<audio controls preload="none" src="https://huggingface.co/TontaubeAI/TontaubeV1/resolve/main/assets/showcase/english-audiobook-03.mp3"></audio>

[Download MP3](assets/showcase/english-audiobook-03.mp3)

**German 1**

<audio controls preload="none" src="https://huggingface.co/TontaubeAI/TontaubeV1/resolve/main/assets/showcase/german-audiobook-01.mp3"></audio>

[Download MP3](assets/showcase/german-audiobook-01.mp3)

### Agentic

**English 1**

<audio controls preload="none" src="https://huggingface.co/TontaubeAI/TontaubeV1/resolve/main/assets/showcase/english-agentic-01.mp3"></audio>

[Download MP3](assets/showcase/english-agentic-01.mp3)

**English 2**

<audio controls preload="none" src="https://huggingface.co/TontaubeAI/TontaubeV1/resolve/main/assets/showcase/english-agentic-02.mp3"></audio>

[Download MP3](assets/showcase/english-agentic-02.mp3)

**English 3**

<audio controls preload="none" src="https://huggingface.co/TontaubeAI/TontaubeV1/resolve/main/assets/showcase/english-agentic-03.mp3"></audio>

[Download MP3](assets/showcase/english-agentic-03.mp3)

## Model architecture

| Predictor | Role | Transformer blocks | Width | Stored parameters |
|---|---|---:|---:|---:|
| CB0 | Semantic audio and duration | 28 | 2,048 | 1,829,116,930 |
| CB1 | Acoustic refinement | 16 | 1,024 | 448,960,512 |
| CB2 | Acoustic refinement | 8 | 1,024 | 327,307,264 |
| CB3 | Acoustic refinement | 4 | 1,024 | 268,577,792 |
| **Total** |  | **56** |  | **2,873,962,498** |

CB0 uses a Qwen3-1.7B-derived transformer trunk. CB1--CB3 use progressively
shallower Qwen3-0.6B-derived trunks. Each predictor replaces the inherited
language-model output layer with a two-layer audio-token head. The four output
streams are decoded with DualCodec; the release inference path then uses the
acoustic encoder and decoder from VibeVoice for continuous reconstruction and
streaming.

## Training

All four predictors were trained exclusively with supervised fine-tuning on
approximately 200,000 hours of paired speech and text across seven languages,
predominantly from public-domain audiobook recordings and openly released
speech corpora.

## Repository contents

```text
TontaubeV1/
├── cb0/
├── cb1/
├── cb2/
└── cb3/
```

Each directory contains its safetensors weights, runtime configuration, and
tokenizer files. The repository does not redistribute DualCodec, W2V-BERT, or
VibeVoice weights. The inference setup downloads those components from their
upstream repositories at pinned revisions.

## Serving performance

With weights resident on one NVIDIA GeForce RTX 5090 and the process warmed,
the streaming path reaches approximately 200 ms to first encoded audio. In
separate non-streaming measurements, end-to-end real-time factor (RTF) is 0.08
for one input and aggregate RTF is 0.02 across eight concurrent inputs. Startup,
model loading, and network latency are excluded.

## Evaluation

The LLM-as-a-judge pairwise audiobook-reading benchmark contains 400 fixed
English passages of 250--500 characters. For each passage, Gemini 3.1 Pro
Preview judges the same output pair twice, once in each presentation order, on
prosody and word-by-word correctness. A TontaubeV1 preference, tie, or
comparator preference scores 1, 0.5, or 0; the figure reports the mean over all
800 order-balanced judgments. A score of 50% denotes parity. Whiskers show 95%
passage-cluster bootstrap intervals; labels beneath the points give tie rates
across individual judge calls.

<p align="center">
  <img src="assets/llm-judge-results.png" alt="TontaubeV1 preference scores against ElevenLabs Flash v2.5, Fish Audio S2 Pro, Gradium, and Cartesia on prosody and correctness.">
</p>

<details>
<summary>Exact preference scores</summary>

| Comparator | Prosody preference | Correctness preference |
|---|---:|---:|
| ElevenLabs Flash v2.5 | 50.1% | 48.9% |
| Fish Audio S2 Pro | 82.1% | 49.6% |
| Gradium API, April 2026 | 86.2% | 54.6% |
| Cartesia Sonic 3 | 82.3% | 60.8% |

</details>

TontaubeV1 uses semantic sampling temperature `0.55` and acoustic temperature
zero in these comparisons. Each waveform is independently normalized to an
average level of -20 dBFS before judging. The technical report gives the
complete judge instructions and further methodological limitations.

Fish Audio uses the same frozen cloning reference as TontaubeV1. ElevenLabs,
Gradium, and Cartesia instead use fixed provider voices while TontaubeV1 uses
the cloning reference. Although voice identity and timbre are excluded from the
rubric, prosody is not fully separable from the reference; this asymmetry may
favor TontaubeV1 in those three comparisons.

On the 1,088 English zero-shot examples of the Seed-TTS evaluation set,
TontaubeV1 obtains 1.66% mean utterance-level word error rate using Whisper
large-v3 transcription at semantic sampling temperature `0.6`.

These automated evaluations measure English reading prosody and text
correctness. They do not establish voice similarity, overall sound quality,
multilingual quality, long-form continuity, or streaming quality.

## Limitations and responsible use

Autoregressive generation can omit, repeat, or alter text and can terminate
too early or too late. Reference conditioning may reproduce incidental
recording properties or imperfectly preserve identity. Quality varies by
language, speaker, style, and text domain. The optional verbalizer can
normalize incorrectly or alter wording. Training emphasized audiobook speech,
so audiobook generation may be more reliable than conversational or agentic
generation.

English and German are the primary languages. German phoneme realization can
be imperfect, and Spanish, French, Italian, Dutch, and Portuguese have not been
evaluated by native speakers; no comparative quality claims are made for those
languages.

Voice cloning can enable impersonation, fraud, non-consensual synthesis, and
misleading media. Obtain permission for reference voices and apply appropriate
authentication, rate limiting, disclosure, logging, and abuse-response
measures in deployed systems. Generated speech must not be treated as verified
identity or factual evidence.

## License and attribution

TontaubeV1 is distributed under the
[Tontaube Community Model License 1.0](https://huggingface.co/TontaubeAI/TontaubeV1/blob/v1.0.0/LICENSE), which is not an open-source
license. The license applies to the Tontaube model materials identified by the
release manifest; third-party material remains subject to its respective
terms. See [Third-Party Notices](THIRD_PARTY_NOTICES.md).

TontaubeV1 builds on Qwen3 and uses the DualCodec speech representation. The
release inference stack also uses components from VibeVoice, W2V-BERT,
PyTorch, Transformers, and vLLM. Their respective licenses and notices remain
applicable.

The generated audio examples in `assets/showcase/` are provided as
demonstration material and are not part of the model materials covered by the
Tontaube Community Model License 1.0.