README.md

---
license: cc-by-nc-4.0
base_model: k2-fsa/OmniVoice
datasets:
  - XXH333/WordVoice-5A
language:
  - en
pipeline_tag: text-to-speech
tags:
  - tts
  - voice-cloning
  - controllable-tts
  - word-level-control
  - omnivoice
---

# OmniVoice Word-Control

[OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) (0.6B masked-diffusion LM TTS on
Qwen3-0.6B) fine-tuned for **five-dimensional word-level acoustic control**, following the
task of [WordVoice (arXiv:2607.06461)](https://huggingface.co/papers/2607.06461) and its
dataset [WordVoice-5A](https://huggingface.co/datasets/XXH333/WordVoice-5A).

Control is injected as **inline special tokens** placed before any word (any subset works;
untagged words are planned freely by the model):

```
<|dur_9|><|bnd_0|><|eng_15|><|pit_17|><|ton_rise|>word
```

| tokens | attribute | discretization |
|---|---|---|
| `<\|dur_0..63\|>` | word duration | 40 ms frames (40 ms – 2.56 s) |
| `<\|bnd_0..4\|>` | pause after word | b0 (none) … b4 (long, >0.4 s) |
| `<\|eng_0..19\|>` | energy | 20 uniform bins over [0, 1] |
| `<\|pit_0..19\|>` | pitch (core F0) | 20 uniform bins over [−1, 1] |
| `<\|ton_flat\|rise\|rrise\|fall\|ffall\|peak\|valley\|>` | pitch contour | 7 morphologies |

## Usage

```python
from omnivoice import OmniVoice  # code: github.com/k2-fsa/OmniVoice

model = OmniVoice.from_pretrained("multimodalart/omnivoice-word-control", device_map="cuda")
audio = model.generate(
    text="I will <|pit_3|><|ton_ffall|>never agree to <|bnd_4|>this!",
    language="en",
    ref_audio="ref.wav",
    ref_text="Transcript of the reference clip.",
)
```

The tokenizer in this repo already contains the 116 control tokens. For strict overall
timing, also pass `duration=` (the diffusion canvas is fixed before generation, so word
duration tokens redistribute time within it). Try it in the
[demo Space](https://huggingface.co/spaces/multimodalart/omnivoice-word-control).

## Training

- Init from `k2-fsa/OmniVoice`; all parameters trained; masked-diffusion objective unchanged
  (control tokens are conditioning only — no loss on text).
- Data: WordVoice-5A **English**, all 4 train shards (~2,138 h, 1.18 M utterances); on-the-fly
  tag construction with dropout (15% utterances untagged / 20% words / attributes kept @ 0.7)
  to preserve plain TTS and enable partial control.
- 40k steps × 8,192 effective tokens (~1 epoch), lr 5e-5 cosine, bf16, single A100-40GB, ≈6 h.
- Final eval loss 4.109 (best of run). Directional probes (hi/lo tag ratio, seed-matched):
  pitch 3.61, energy 2.52, duration 2.70.

Fine-tuning recipe: the `word-control` patch set on the OmniVoice trainer
(`omnivoice/data/word_control.py` — tag vocabulary, binning, alignment, dropout).

## License & lineage

Weights are **CC-BY-NC-4.0** (derivative of the CC-BY-NC OmniVoice release).
Data: WordVoice-5A (CC-BY-4.0). Task formulation: WordVoice, arXiv:2607.06461.