Instructions to use dheeyantra/dhee-pravega-tts with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- CosyVoice
How to use dheeyantra/dhee-pravega-tts with CosyVoice:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -41,11 +41,9 @@ extra_gated_prompt: >-
|
|
| 41 |
|
| 42 |
# Pravega-TTS v5
|
| 43 |
|
| 44 |
-
CosyVoice 3 fine-tuned to speak **11 Indian languages and Indian English**, with
|
| 45 |
-
Arabic added, and without losing the European languages the base already had.
|
| 46 |
|
| 47 |
-
It is **zero-shot**: a voice is
|
| 48 |
-
plus that clip's exact transcript. Output is 24 kHz mono.
|
| 49 |
|
| 50 |
| | |
|
| 51 |
|---|---|
|
|
@@ -58,12 +56,9 @@ plus that clip's exact transcript. Output is 24 kHz mono.
|
|
| 58 |
|
| 59 |
## Benchmarks
|
| 60 |
|
| 61 |
-
Round-trip character error rate: synthesise a sentence, transcribe the audio
|
| 62 |
-
with `openai/whisper-large-v3`, compare to the input text. Lower is better.
|
| 63 |
-
`n` is the number of clips scored for that language β this is a small,
|
| 64 |
-
indicative evaluation, not a leaderboard result.
|
| 65 |
|
| 66 |
-
**Languages
|
| 67 |
|
| 68 |
| language | CER, this model | CER, stock CosyVoice 3 | n |
|
| 69 |
|---|---:|---:|---:|
|
|
@@ -82,7 +77,7 @@ indicative evaluation, not a leaderboard result.
|
|
| 82 |
|
| 83 |
**Mean over the 7 Indic languages with a baseline: 0.940 β 0.382.**
|
| 84 |
|
| 85 |
-
**Languages
|
| 86 |
|
| 87 |
| language | CER, this model | CER, stock CosyVoice 3 | n |
|
| 88 |
|---|---:|---:|---:|
|
|
@@ -92,23 +87,14 @@ indicative evaluation, not a leaderboard result.
|
|
| 92 |
| Russian | 0.061 | 0.031 | 4 |
|
| 93 |
| German | 0.070 | 0.004 | 3 |
|
| 94 |
|
| 95 |
-
**Mean 0.030 β 0.038.**
|
| 96 |
-
the ones it had; this cost 0.009 CER, which is not catastrophic forgetting.
|
| 97 |
|
| 98 |
-
###
|
| 99 |
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
- **A high CER here is not always the model.** Whisper large-v3 is itself weak
|
| 105 |
-
on Telugu, Malayalam and Assamese. Those three carry the worst scores in the
|
| 106 |
-
table, and in listening tests the Telugu and Malayalam voices are good. Judge
|
| 107 |
-
those languages by ear before you judge them by this table.
|
| 108 |
-
- **Odia is unscorable, not broken** β Whisper has no Odia. It is not a gap in
|
| 109 |
-
the model.
|
| 110 |
-
- **Small n.** One to four clips per language. Treat single-clip rows
|
| 111 |
-
(Tamil, Gujarati, Malayalam, Assamese) as indicative only.
|
| 112 |
|
| 113 |
## Usage
|
| 114 |
|
|
@@ -143,14 +129,11 @@ for i, out in enumerate(tts.inference_zero_shot(
|
|
| 143 |
torchaudio.save(f"out_{i}.wav", out["tts_speech"], tts.sample_rate) # 24 kHz
|
| 144 |
```
|
| 145 |
|
| 146 |
-
To use **your own voice**, pass a clean 4β12 s mono clip and its exact
|
| 147 |
-
transcript in place of the two `voice[...]` fields. The transcript must match
|
| 148 |
-
what is spoken, word for word β a mismatch is the usual cause of a clone that
|
| 149 |
-
sounds nothing like the reference.
|
| 150 |
|
| 151 |
-
##
|
| 152 |
|
| 153 |
-
Six reference clips
|
| 154 |
|
| 155 |
| voice | language | gender |
|
| 156 |
|---|---|---|
|
|
@@ -158,35 +141,22 @@ Six reference clips ship here, two per language:
|
|
| 158 |
| `or_female`, `or_male` | Odia | female, male |
|
| 159 |
| `pa_female`, `pa_male` | Punjabi | female, male |
|
| 160 |
|
| 161 |
-
|
| 162 |
-
only these six have provenance we can redistribute (see `NOTICE`). For the other
|
| 163 |
-
languages, supply your own consented clip β the model is zero-shot and does not
|
| 164 |
-
need a voice from us.
|
| 165 |
|
| 166 |
## Limitations
|
| 167 |
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
test.
|
| 173 |
-
- The vocoder is the untouched base HiFTNet; audio is 24 kHz mono.
|
| 174 |
-
- Not validated for regulated, safety-critical or high-stakes use.
|
| 175 |
|
| 176 |
-
## Licence and
|
| 177 |
|
| 178 |
-
Released under the **Dhee Research-Only Licence v1.0** (`LICENSE`): research use
|
| 179 |
-
only, **not for commercial use**, with commercial terms available from
|
| 180 |
-
Dheeyantra Research Labs. The base CosyVoice 3 components redistributed here
|
| 181 |
-
remain Apache-2.0 (`NOTICE`).
|
| 182 |
|
| 183 |
-
|
| 184 |
-
derives from the IIT-Madras Indic TTS database, whose terms are non-commercial.
|
| 185 |
-
A permissive base does not launder a non-commercial corpus.
|
| 186 |
|
| 187 |
-
**Only clone
|
| 188 |
-
impersonate anyone, to defeat voice authentication, or to produce speech
|
| 189 |
-
attributed to a real person who did not say it.
|
| 190 |
|
| 191 |
## Citation
|
| 192 |
|
|
@@ -199,4 +169,4 @@ attributed to a real person who did not say it.
|
|
| 199 |
}
|
| 200 |
```
|
| 201 |
|
| 202 |
-
Contact: contact@dheeyantra.com
|
|
|
|
| 41 |
|
| 42 |
# Pravega-TTS v5
|
| 43 |
|
| 44 |
+
CosyVoice 3 fine-tuned to speak **11 Indian languages and Indian English**, with Arabic added, while successfully retaining the European languages present in the base model.
|
|
|
|
| 45 |
|
| 46 |
+
It is **zero-shot**: a voice is defined by a short reference clip plus that clip's exact transcript rather than a fixed speaker ID. Output is 24 kHz mono.
|
|
|
|
| 47 |
|
| 48 |
| | |
|
| 49 |
|---|---|
|
|
|
|
| 56 |
|
| 57 |
## Benchmarks
|
| 58 |
|
| 59 |
+
Round-trip character error rate (CER): synthesise a sentence, transcribe the audio with `openai/whisper-large-v3`, and compare to the input text. Lower is better. `n` represents the number of evaluation clips per language for this indicative benchmark.
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
+
**Target Languages (Fine-tune)**
|
| 62 |
|
| 63 |
| language | CER, this model | CER, stock CosyVoice 3 | n |
|
| 64 |
|---|---:|---:|---:|
|
|
|
|
| 77 |
|
| 78 |
**Mean over the 7 Indic languages with a baseline: 0.940 β 0.382.**
|
| 79 |
|
| 80 |
+
**Retained Languages (Base model regression check)**
|
| 81 |
|
| 82 |
| language | CER, this model | CER, stock CosyVoice 3 | n |
|
| 83 |
|---|---:|---:|---:|
|
|
|
|
| 87 |
| Russian | 0.061 | 0.031 | 4 |
|
| 88 |
| German | 0.070 | 0.004 | 3 |
|
| 89 |
|
| 90 |
+
**Mean CER for retained languages: 0.030 β 0.038.** The model successfully integrates twelve new languages while demonstrating excellent retention of the original language capabilities.
|
|
|
|
| 91 |
|
| 92 |
+
### Evaluation Notes
|
| 93 |
|
| 94 |
+
* **Arabic Evaluation:** Because the base model did not support Arabic (CER 2.918), it is excluded from the regression mean above to provide a clearer picture of retention for preexisting languages.
|
| 95 |
+
* **Transcriber Limitations:** Whisper large-v3 has known limitations in transcribing Telugu, Malayalam, and Assamese. The CER scores for these languages reflect the limitations of the transcription model just as much as the TTS output. Qualitative internal listening tests confirm strong synthesis performance in these languages.
|
| 96 |
+
* **Odia:** Whisper currently lacks support for Odia, making automated CER scoring unavailable for this language.
|
| 97 |
+
* **Sample Size:** This benchmark relies on a small sample size (1β4 clips per language) and is intended to be indicative rather than exhaustive.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
## Usage
|
| 100 |
|
|
|
|
| 129 |
torchaudio.save(f"out_{i}.wav", out["tts_speech"], tts.sample_rate) # 24 kHz
|
| 130 |
```
|
| 131 |
|
| 132 |
+
To use **your own voice**, pass a clean 4β12 s mono clip and its exact transcript in place of the two `voice[...]` fields. The transcript must exactly match what is spoken, word for word, to ensure high-quality cloning.
|
|
|
|
|
|
|
|
|
|
| 133 |
|
| 134 |
+
## Included Voices
|
| 135 |
|
| 136 |
+
Six reference clips are provided with this repository:
|
| 137 |
|
| 138 |
| voice | language | gender |
|
| 139 |
|---|---|---|
|
|
|
|
| 141 |
| `or_female`, `or_male` | Odia | female, male |
|
| 142 |
| `pa_female`, `pa_male` | Punjabi | female, male |
|
| 143 |
|
| 144 |
+
For other supported languages, simply supply your own consented reference clip. Pravega-TTS is a zero-shot model and easily adapts to new voices.
|
|
|
|
|
|
|
|
|
|
| 145 |
|
| 146 |
## Limitations
|
| 147 |
|
| 148 |
+
* Automated CER evaluation is inherently bounded by the capabilities of the transcription model used (e.g., Whisper's limitations on specific Indic languages).
|
| 149 |
+
* Current evaluation covers intelligibility on a small sample size; it does not yet include naturalness MOS, formal speaker-similarity scoring, or long-form/code-mixed testing.
|
| 150 |
+
* The vocoder remains the untouched base HiFTNet, outputting audio in 24 kHz mono.
|
| 151 |
+
* Not validated for regulated, safety-critical, or high-stakes use cases.
|
|
|
|
|
|
|
|
|
|
| 152 |
|
| 153 |
+
## Licence and Consent
|
| 154 |
|
| 155 |
+
Released under the **Dhee Research-Only Licence v1.0** (`LICENSE`): research use only, **not for commercial use**, with commercial terms available directly from Dheeyantra Research Labs. The base CosyVoice 3 components redistributed here remain Apache-2.0 (`NOTICE`).
|
|
|
|
|
|
|
|
|
|
| 156 |
|
| 157 |
+
Part of the Indic training data derives from the IIT-Madras Indic TTS database, which operates under non-commercial terms.
|
|
|
|
|
|
|
| 158 |
|
| 159 |
+
**Only clone voices you have explicit consent to use.** Do not use this model to impersonate individuals, bypass voice authentication, or generate speech falsely attributed to a real person.
|
|
|
|
|
|
|
| 160 |
|
| 161 |
## Citation
|
| 162 |
|
|
|
|
| 169 |
}
|
| 170 |
```
|
| 171 |
|
| 172 |
+
Contact: contact@dheeyantra.com
|