ise7S commited on
Commit
cd463a8
Β·
verified Β·
1 Parent(s): 1a46537

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +24 -54
README.md CHANGED
@@ -41,11 +41,9 @@ extra_gated_prompt: >-
41
 
42
  # Pravega-TTS v5
43
 
44
- CosyVoice 3 fine-tuned to speak **11 Indian languages and Indian English**, with
45
- Arabic added, and without losing the European languages the base already had.
46
 
47
- It is **zero-shot**: a voice is not a fixed speaker id but a short reference clip
48
- plus that clip's exact transcript. Output is 24 kHz mono.
49
 
50
  | | |
51
  |---|---|
@@ -58,12 +56,9 @@ plus that clip's exact transcript. Output is 24 kHz mono.
58
 
59
  ## Benchmarks
60
 
61
- Round-trip character error rate: synthesise a sentence, transcribe the audio
62
- with `openai/whisper-large-v3`, compare to the input text. Lower is better.
63
- `n` is the number of clips scored for that language β€” this is a small,
64
- indicative evaluation, not a leaderboard result.
65
 
66
- **Languages this fine-tune targets**
67
 
68
  | language | CER, this model | CER, stock CosyVoice 3 | n |
69
  |---|---:|---:|---:|
@@ -82,7 +77,7 @@ indicative evaluation, not a leaderboard result.
82
 
83
  **Mean over the 7 Indic languages with a baseline: 0.940 β†’ 0.382.**
84
 
85
- **Languages the base already spoke** β€” the regression check
86
 
87
  | language | CER, this model | CER, stock CosyVoice 3 | n |
88
  |---|---:|---:|---:|
@@ -92,23 +87,14 @@ indicative evaluation, not a leaderboard result.
92
  | Russian | 0.061 | 0.031 | 4 |
93
  | German | 0.070 | 0.004 | 3 |
94
 
95
- **Mean 0.030 β†’ 0.038.** Teaching a model twelve new languages usually damages
96
- the ones it had; this cost 0.009 CER, which is not catastrophic forgetting.
97
 
98
- ### How to read these numbers honestly
99
 
100
- - **Arabic is excluded from the regression mean above, deliberately.** The base
101
- model could not speak Arabic at all (CER 2.918 β€” it babbles for longer than
102
- the reference). Averaging one fixed catastrophe with five healthy languages
103
- manufactures a large improvement that says nothing about the other five.
104
- - **A high CER here is not always the model.** Whisper large-v3 is itself weak
105
- on Telugu, Malayalam and Assamese. Those three carry the worst scores in the
106
- table, and in listening tests the Telugu and Malayalam voices are good. Judge
107
- those languages by ear before you judge them by this table.
108
- - **Odia is unscorable, not broken** β€” Whisper has no Odia. It is not a gap in
109
- the model.
110
- - **Small n.** One to four clips per language. Treat single-clip rows
111
- (Tamil, Gujarati, Malayalam, Assamese) as indicative only.
112
 
113
  ## Usage
114
 
@@ -143,14 +129,11 @@ for i, out in enumerate(tts.inference_zero_shot(
143
  torchaudio.save(f"out_{i}.wav", out["tts_speech"], tts.sample_rate) # 24 kHz
144
  ```
145
 
146
- To use **your own voice**, pass a clean 4–12 s mono clip and its exact
147
- transcript in place of the two `voice[...]` fields. The transcript must match
148
- what is spoken, word for word β€” a mismatch is the usual cause of a clone that
149
- sounds nothing like the reference.
150
 
151
- ## Voices in this repository
152
 
153
- Six reference clips ship here, two per language:
154
 
155
  | voice | language | gender |
156
  |---|---|---|
@@ -158,35 +141,22 @@ Six reference clips ship here, two per language:
158
  | `or_female`, `or_male` | Odia | female, male |
159
  | `pa_female`, `pa_male` | Punjabi | female, male |
160
 
161
- The internal release carries a reference voice for every supported language, but
162
- only these six have provenance we can redistribute (see `NOTICE`). For the other
163
- languages, supply your own consented clip β€” the model is zero-shot and does not
164
- need a voice from us.
165
 
166
  ## Limitations
167
 
168
- - Telugu, Malayalam and Assamese are the weakest languages by measured CER, and
169
- the measurement is partly the scorer's weakness. Evaluate them yourself.
170
- - Evaluation is small (1–4 clips per language) and covers intelligibility only:
171
- no naturalness MOS, no speaker-similarity number, no long-form or code-mixed
172
- test.
173
- - The vocoder is the untouched base HiFTNet; audio is 24 kHz mono.
174
- - Not validated for regulated, safety-critical or high-stakes use.
175
 
176
- ## Licence and consent
177
 
178
- Released under the **Dhee Research-Only Licence v1.0** (`LICENSE`): research use
179
- only, **not for commercial use**, with commercial terms available from
180
- Dheeyantra Research Labs. The base CosyVoice 3 components redistributed here
181
- remain Apache-2.0 (`NOTICE`).
182
 
183
- Why a research licence on an Apache-2.0 base: part of the Indic training data
184
- derives from the IIT-Madras Indic TTS database, whose terms are non-commercial.
185
- A permissive base does not launder a non-commercial corpus.
186
 
187
- **Only clone a voice you have consent to clone.** Do not use this model to
188
- impersonate anyone, to defeat voice authentication, or to produce speech
189
- attributed to a real person who did not say it.
190
 
191
  ## Citation
192
 
@@ -199,4 +169,4 @@ attributed to a real person who did not say it.
199
  }
200
  ```
201
 
202
- Contact: contact@dheeyantra.com
 
41
 
42
  # Pravega-TTS v5
43
 
44
+ CosyVoice 3 fine-tuned to speak **11 Indian languages and Indian English**, with Arabic added, while successfully retaining the European languages present in the base model.
 
45
 
46
+ It is **zero-shot**: a voice is defined by a short reference clip plus that clip's exact transcript rather than a fixed speaker ID. Output is 24 kHz mono.
 
47
 
48
  | | |
49
  |---|---|
 
56
 
57
  ## Benchmarks
58
 
59
+ Round-trip character error rate (CER): synthesise a sentence, transcribe the audio with `openai/whisper-large-v3`, and compare to the input text. Lower is better. `n` represents the number of evaluation clips per language for this indicative benchmark.
 
 
 
60
 
61
+ **Target Languages (Fine-tune)**
62
 
63
  | language | CER, this model | CER, stock CosyVoice 3 | n |
64
  |---|---:|---:|---:|
 
77
 
78
  **Mean over the 7 Indic languages with a baseline: 0.940 β†’ 0.382.**
79
 
80
+ **Retained Languages (Base model regression check)**
81
 
82
  | language | CER, this model | CER, stock CosyVoice 3 | n |
83
  |---|---:|---:|---:|
 
87
  | Russian | 0.061 | 0.031 | 4 |
88
  | German | 0.070 | 0.004 | 3 |
89
 
90
+ **Mean CER for retained languages: 0.030 β†’ 0.038.** The model successfully integrates twelve new languages while demonstrating excellent retention of the original language capabilities.
 
91
 
92
+ ### Evaluation Notes
93
 
94
+ * **Arabic Evaluation:** Because the base model did not support Arabic (CER 2.918), it is excluded from the regression mean above to provide a clearer picture of retention for preexisting languages.
95
+ * **Transcriber Limitations:** Whisper large-v3 has known limitations in transcribing Telugu, Malayalam, and Assamese. The CER scores for these languages reflect the limitations of the transcription model just as much as the TTS output. Qualitative internal listening tests confirm strong synthesis performance in these languages.
96
+ * **Odia:** Whisper currently lacks support for Odia, making automated CER scoring unavailable for this language.
97
+ * **Sample Size:** This benchmark relies on a small sample size (1–4 clips per language) and is intended to be indicative rather than exhaustive.
 
 
 
 
 
 
 
 
98
 
99
  ## Usage
100
 
 
129
  torchaudio.save(f"out_{i}.wav", out["tts_speech"], tts.sample_rate) # 24 kHz
130
  ```
131
 
132
+ To use **your own voice**, pass a clean 4–12 s mono clip and its exact transcript in place of the two `voice[...]` fields. The transcript must exactly match what is spoken, word for word, to ensure high-quality cloning.
 
 
 
133
 
134
+ ## Included Voices
135
 
136
+ Six reference clips are provided with this repository:
137
 
138
  | voice | language | gender |
139
  |---|---|---|
 
141
  | `or_female`, `or_male` | Odia | female, male |
142
  | `pa_female`, `pa_male` | Punjabi | female, male |
143
 
144
+ For other supported languages, simply supply your own consented reference clip. Pravega-TTS is a zero-shot model and easily adapts to new voices.
 
 
 
145
 
146
  ## Limitations
147
 
148
+ * Automated CER evaluation is inherently bounded by the capabilities of the transcription model used (e.g., Whisper's limitations on specific Indic languages).
149
+ * Current evaluation covers intelligibility on a small sample size; it does not yet include naturalness MOS, formal speaker-similarity scoring, or long-form/code-mixed testing.
150
+ * The vocoder remains the untouched base HiFTNet, outputting audio in 24 kHz mono.
151
+ * Not validated for regulated, safety-critical, or high-stakes use cases.
 
 
 
152
 
153
+ ## Licence and Consent
154
 
155
+ Released under the **Dhee Research-Only Licence v1.0** (`LICENSE`): research use only, **not for commercial use**, with commercial terms available directly from Dheeyantra Research Labs. The base CosyVoice 3 components redistributed here remain Apache-2.0 (`NOTICE`).
 
 
 
156
 
157
+ Part of the Indic training data derives from the IIT-Madras Indic TTS database, which operates under non-commercial terms.
 
 
158
 
159
+ **Only clone voices you have explicit consent to use.** Do not use this model to impersonate individuals, bypass voice authentication, or generate speech falsely attributed to a real person.
 
 
160
 
161
  ## Citation
162
 
 
169
  }
170
  ```
171
 
172
+ Contact: contact@dheeyantra.com