eduardem

/

piper-tts-romanian

Like 15

@MarusGradinaru Checkpoints are already in the repo: https://huggingface.co/eduardem/piper-tts-romanian/resolve/main/voices/sanda/ro_RO-sanda-medium.ckpt

But, there might be some issues with your plan.

1. Stack mismatch. export_onnx_streaming only exists in the archived rhasspy/piper. Sanda was trained on piper1-gpl v1.4.1 (Lightning 2.6.1), not the old stack. I expect it to load anyway the VITS modules are effectively unchanged between the two, the model_g/model_d prefixes match, and hparams["dataset"] is None, which is what load_from_checkpoint wants but I haven't run the streaming exporter against a Sanda checkpoint, so treat that as "should work", not "tested". If it doesn't work on a key, strict=False is the first thing to try.

2. espeak-ng version. Paroli pins piper-phonemize-2023-11-14. Its ro_dict is byte-identical to the one Sanda was trained with (md5 5c9530ca4d4f92a3265d15a9e7a23f2c), so word - phoneme lookups match.
   But the shared tables (phondata, phonindex, phontab, intonations) differ between that build and piper1-gpl's. If the output sounds off compared to the monolithic ONNX, diff espeak-ng -v ro --ipa between the two builds before suspecting the model.

Also, would be great if you could share the results. I am interested in:

• is ti working with Paroli's 2023 espeak ?
• Any audible changes in Romanian phoneme output versus current piper1-gpl ?
• Does the encoder/decoder split reproduces the monolithic ONNX ?

Also you might want to check my latest voice, Liana, it is the best voice that I made. It can read entire books without halucinating and it pronounce correctly the English words borrowed in Romanian: https://huggingface.co/eduardem/piper-liana-romanian

Wow ! That is great ! I don't know why I didn't see the checkpoints in the first place... because I searched really thoroughly ! Anyway, it's good that I found them ! I will start the tests and keep you updated on what happens. Thank you very much !

About Liana... Now, there really is no checkpoint in Liana's "voices" folder:
https://huggingface.co/eduardem/piper-liana-romanian/tree/main/voices/liana-medium

By the way, are you Romanian too? Like I am... 😃

Yes, I have not yet published the checkpoints for Liana, I have some issues with the licensing, clarifying now if it allows it.

Yes, I am Romanian, I doubt a foreigner will be active in creating Romanian voices :D

Can we talk in romanian here, or is not allowed ? 😀 It's my first time on Huggingface. I sign-up just to talk with you...

I managed to adapt the voices for the NanoPi's NPU. Here's the tutorial and some samples:
https://www.elforum.info/topic/165890-avem-pe-aici-fani-radxa-friendlyelec-sau-alte-sbc-uri-in-afara-de-raspberry-pi/?do=findComment&comment=2336363
But I don't really like how any of those voices sound on my NPU and the texts I tried. They all have different errors. 😟

Edit:
It seems you can't see the tutorial in the forum attachment unless you're a member there. So I uploaded the document to my website:
https://marus-gradinaru.42web.io/NanoPi-M5/How-to-add-voice-to-your-NanoPi-M5.docx

I don't think it is allowed to write Romanian, their bots needs to crawl all this conversation.

Let me try to do that for Liana and see. But again the problem can be related to the different espeak versions

I don't understand... You said that you want to try what I did in my tutorial, with Liana ? That is for NanoPi M5 board. Do you have one ?

I made another test... I used the voices as you intended to, with the piper1-gpl using CPU on my laptop, and the texts I got errors (like for lili-2 and lili-3). The big errors ar still there, but the voice improved a little...

Edit:
By the way, here, Lili-medium is seriously hallucinating ! Haven't you noticed ? Or does it only happen to me...?
https://huggingface.co/eduardem/piper-tts-romanian#long-text-continuous-speech-no-hallucinations

All issues related from the previous generation of languages (hallucinations for texts longer than one phrase, English borrowed words pronunciation, Romanian and English names, numbers, etc) are fixed in Liana voice that is 4rd generation voice trained over 4 months and more than 100 iterations.

Please drop me an email at eduardem@flpf.party I will share the Liana medium checkpoints with you

I sent you my email at the specified address. It begins with the same name as here...

I did a little more research with AI and found that some of the artifacts generated are caused by the padding that Paroli applies to chunks smaller than 55 frames. I'll let him explain ... 😀

We found that the RKNN decoder itself is not the main source of the audio artifacts.

Using exactly the same 55-frame input tensors, the ONNX CPU decoder and the RKNN NPU decoder produce almost identical waveforms:

Correlation: 0.999994
RMSE: 8.26
Max error: 67

The real problem appears when Paroli has a chunk shorter than 55 frames. The RKNN model was compiled with a fixed input size of 55 frames, so Paroli pads shorter chunks with zeros before inference and crops the output afterwards.

We reproduced this behavior with the ONNX decoder itself:

A: ONNX with the original 50 frames
B: ONNX with 50 frames + 5 zero-padded frames, then cropped
C: RKNN with the same 50→55 padding

Results:

A vs B:
Correlation: 0.998799
RMSE: 103.1
Max error: 1569

B vs C:
Correlation: 0.999996
RMSE: 6.47
Max error: 44

So RKNN reproduces the padded ONNX result extremely well. The much larger difference is caused by the zero-padding itself, presumably because the decoder/vocoder's convolutional receptive field allows the padded future frames to influence the preceding waveform.

Our next step is therefore to modify Paroli's chunking so that RKNN always receives 55 real frames whenever possible, especially at the beginning and end of a sequence, instead of padding incomplete chunks with zeros.