Create README.md

README.md

@@ -0,0 +1,140 @@
+---
+language:
+  - et
+license: mit
+base_model: sarulab-speech/UTMOSv2
+tags:
+  - audio
+  - speech
+  - mos-prediction
+  - speech-quality-assessment
+  - text-to-speech
+  - estonian
+  - low-resource
+---
+
+# UTMOSv2-Estonian
+
+[UTMOSv2](https://github.com/sarulab-speech/UTMOSv2) (Baba et al., 2024) fine-tuned
+on Estonian listening-test data to predict the naturalness mean opinion score
+(MOS) of synthesized speech.
+
+The mean opinion score quantifies the naturalness of synthesized speech by
+averaging subjective ratings from multiple listeners, and is the standard measure
+in speech synthesis evaluation. Obtaining it requires a listening test, which is
+labour-intensive and time-consuming — a cost that falls hardest on languages with
+few speakers, where recruiting sufficient raters is itself difficult.
+
+**[Interactive demo](https://huggingface.co/spaces/monatolmats/utmosv2-estonian-demo)**
+
+## Scope of this work
+
+The architecture and the English pretraining are the work of the UTMOSv2 authors.
+The contribution here is the fine-tuning on Estonian data and its evaluation on
+Estonian and Võro, carried out for the bachelor's thesis *Automatic Speech
+Synthesis Quality Assessment for Finno-Ugric Languages* (Mona Tolmats, University
+of Tartu, 2025; supervisor Liisa Rätsep).
+
+## Usage
+
+UTMOSv2 conditions on a data-domain one-hot vector and cannot derive one for a
+corpus it has not seen. During fine-tuning the Estonian data occupied three of
+UTMOSv2's existing domain slots. Following the recommendation of the UTMOSv2
+authors, each clip should be scored under all three and the results averaged.
+**Scoring under a single arbitrary domain will produce systematically different
+values.**
+
+```bash
+pip install git+https://github.com/sarulab-speech/UTMOSv2.git
+huggingface-cli download monatolmats/utmosv2-estonian --local-dir weights
+```
+
+```python
+import utmosv2
+
+model = utmosv2.create_model(
+    config="fusion_stage3",
+    checkpoint_path="weights/utmosv2_estonian.pth",
+)
+
+DOMAINS = ["somos", "blizzard2010-ES3", "blizzard2010-ES1"]
+
+runs = [model.predict(input_dir="wavs/", predict_dataset=d) for d in DOMAINS]
+for rows in zip(*runs):
+    mos = sum(r["predicted_mos"] for r in rows) / len(rows)
+    print(f"{mos:.2f}  {rows[0]['file_path']}")
+```
+
+Input should be 16 kHz mono audio; output is a predicted MOS on the 1–5 scale.
+
+## Model details
+
+| | |
+| --- | --- |
+| Architecture | UTMOSv2 `fusion_stage3`: a wav2vec 2.0 branch and four EfficientNetV2-S mel-spectrogram branches, fused with a data-domain embedding |
+| Initialised from | `fold0_s42_best_model.pth`, a single fold of the released 5-fold ensemble |
+| Optimiser | AdamW, learning rate 1e-4, cosine annealing |
+| Batch size | 4 |
+| Input length | first 10 s of each clip |
+| Stopping | epoch 33, on validation MSE |
+| Hardware | NVIDIA Tesla A100 40 GB, University of Tartu HPC cluster |
+
+## Training data
+
+Listening-test results from evaluation campaigns run at the University of Tartu
+(Rätsep et al.). Fine-tuning used the three Estonian campaigns; the Võro campaign
+was held out entirely to assess cross-lingual transfer.
+
+| Year | Language | Systems | Clips | Ratings | Role |
+| --- | --- | --- | --- | --- | --- |
+| 2020 | Estonian | 5 | 850 | 17 000 | fine-tuning |
+| 2022 | Estonian | 7 | 1 400 | 5 600 | fine-tuning |
+| 2024 | Estonian | 9 | 2 560 | 12 800 | fine-tuning |
+| 2023 | Võro | 8 | 800 | 2 600 | held-out test |
+
+The 4 810 Estonian clips were split 90/10 into 4 329 training and 481 validation,
+stratified by MOS and by source campaign. Ratings were averaged per clip, so
+targets are continuous. The distribution is skewed toward mid and high scores and
+was left unbalanced, to reflect the composition of real listening tests.
+
+The data is not released; it belongs to the projects cited in the thesis.
+
+## Evaluation
+
+Estonian validation set:
+
+| Model | MSE ↓ | LCC ↑ | SRCC ↑ | KTAU ↑ |
+| --- | --- | --- | --- | --- |
+| wav2vec 2.0 | 0.352 | 0.669 | 0.630 | 0.464 |
+| SCOREQ | **0.224** | **0.802** | 0.768 | 0.589 |
+| UTMOSv2-Estonian | 0.230 | 0.797 | **0.784** | **0.600** |
+
+Võro test set, unseen during fine-tuning:
+
+| Model | MSE ↓ | LCC ↑ | SRCC ↑ | KTAU ↑ |
+| --- | --- | --- | --- | --- |
+| wav2vec 2.0 | 0.639 | 0.177 | 0.168 | 0.118 |
+| SCOREQ | 0.632 | 0.202 | 0.191 | 0.132 |
+| UTMOSv2-Estonian | **0.606** | **0.349** | **0.311** | **0.220** |
+
+Utterance-level correlation on Võro is considerably lower than on Estonian. At
+the system level the picture is better: of the three models compared, this was
+the only one to rank all eight Võro synthesis systems in the same order as human
+raters, and the only one whose 95 % confidence intervals overlapped the human
+intervals for every system.
+
+## Limitations
+
+- **Intended for ranking, not certification.** Like MOS predictors generally,
+  this model is suited to ordering systems rather than grading individual clips.
+  Differences below 0.2 MOS on a single utterance fall within noise; averaging
+  over at least 20 utterances is advisable when comparing systems.
+- **Predictions are not deterministic.** The spectrogram branch samples random
+  windows, so repeated scoring of the same file varies slightly.
+- **Cross-lingual transfer is demonstrated only for Võro**, which is closely
+  related to Estonian. No claim is made about more distant Finno-Ugric languages.
+- **Narrow rating range in the training data.** Very low scores are almost absent,
+  so discrimination between good and excellent systems is weaker than between
+  poor and good ones.
+- **Single fold**, not the 5-fold ensemble of the released UTMOSv2, so variance is
+  higher than the published UTMOSv2 figures.