---
language:
- en
license: apache-2.0
tags:
- text-to-speech
- tts
- matcha-tts
- flow-matching
- tflite
- onnx
- audio
- on-device
datasets:
- ljspeech
metrics:
- loss
- wer
pipeline_tag: text-to-speech
---

# Sonora (Phase 0)

**Sonora** is a lightweight, non-autoregressive text-to-speech (TTS) model designed for real-time, on-device, and resource-constrained environments. 

This model card covers the **Phase 0 baseline** fine-tune, which is built on the **Matcha-TTS** architecture and fine-tuned on the single-speaker LJSpeech dataset. The final model graph embeds the **HiFi-GAN vocoder** (`hifigan_T2_v1`) to execute full end-to-end waveform synthesis directly in a single forward pass (Text IDs $\rightarrow$ PCM Audio).

* **Voice Engine Integration:** Deployed inside **Project Prosodia** (`ProsodiaActor`)
* **Underlying Architecture:** Optimal Transport Conditional Flow Matching (OT-CFM)
* **Parameter Count:** ~18.2 Million
* **Sample Rate:** 22,050 Hz (mono)

---

## Model Details

* **Developed by:** Artificial Humanity
* **Model Type:** Acoustic Flow-Matching Model + HiFi-GAN Vocoder (end-to-end graph)
* **Language(s):** English (en)
* **License:** Apache 2.0
* **Base Checkpoint:** `matcha_ljspeech.ckpt` (v1.0)
* **Parent Model Paper:** [Matcha-TTS: A Fast CFM Text-to-Speech Model (Mehta et al., 2024)](https://arxiv.org/abs/2309.03199)

---

## Active Formats

The model is distributed in multiple formats for execution on different hardware backends:

1. **`checkpoint_epoch=199_e2e_float32.tflite`** (186.9 MB) — **Recommended / verified.** High-precision Float32 model for desktop CPU/GPU and on-device execution via LiteRT. Numerically verified against the ONNX oracle (deterministic waveform cosine = 1.0000). Clean `f32` I/O — drop-in for the Prosodia Rust actor contract.
2. **`checkpoint_epoch=199_e2e_fp16weights_f32io.tflite`** (59.4 MB) — **Recommended for mobile/edge.** Weights-only Float16 with **f32 I/O tensors** (TF post-training float16 quantization of the corrected graph, tensor names restored to the engine contract via `Sonora/scripts/rename_tflite_tensors.py`). Drop-in for the Prosodia actor: deterministic ONNX-parity cosine 0.9991, ASR parity with the ONNX oracle.
3. **`checkpoint_epoch=199_e2e_float16.tflite`** (94.1 MB) — onnx2tf's Float16 build. **Caveat:** its `scales`/`wav` I/O tensors are `f16`, so it will not allocate on the plain CPU interpreter and the current Prosodia actor (which reads/writes those as `f32`) cannot use it as-is — kept for delegate experiments; prefer the `fp16weights_f32io` artifact above.
4. **`checkpoint_epoch=199_e2e.onnx`** (118.7 MB) — Standalone ONNX graph; the numerical **oracle** used by the export-fidelity gate (`Sonora/scripts/export_fidelity_referee.py`).
5. **`v1-ljspeech/litert-split/`** — **LiteRT split-graph lane (mobile/delegate-ready).** The same Epoch-199 weights re-authored to fixed shapes (256 phonemes / 512 mel frames, runtime float mask) and converted per-module with `litert-torch` via the [litert-samples](https://github.com/google-ai-edge/litert-samples) Matcha recipe: `matcha_textenc_fp16.tflite` (14.8 MB, in: emb+mask → mu, logw), `matcha_decoder_fp16.tflite` (22.5 MB, CFM vector field), `matcha_vocoder_fp16.tflite` (29.0 MB), plus `emb.bin` (178×192 f32 phoneme table, host lookup) and `config.json` (symbols, mel stats, shapes). Host orchestrates: embed/intersperse/pad → textenc → durations/length-regulate → Euler ODE ×N over the decoder → vocoder. **Verified:** per-graph tflite-vs-torch corr 1.000000, end-to-end fp16 waveform corr ≥0.9996 vs torch, all graphs GPU-clean; `sample_*.wav` are the parity renders (human-auditioned). Delegate placement per the upstream Pixel 8a findings: decoder on CPU, textenc + vocoder on GPU (RTF ~0.8). Exposes `logw` to the host, enabling per-token duration control that the monolithic e2e graphs cannot offer.

> ### ⚠️ Revision note — 2026-07-12: TFLite artifacts re-exported (fidelity fix)
> The original TFLite exports (through 2026-07-11) were **numerically broken** — they produced
> speech-shaped garble while the ONNX rendered cleanly. Root cause: the Matcha text-encoder's
> channel-axis (`dim=1`) LayerNorm was mislowered by `onnx2tf`'s NHWC layout pass (3D-tensor axis
> ambiguity), corrupting the encoder; the decoder and vocoder converted correctly. Fixed upstream
> in the [Sonora training repo](https://github.com/Artificial-Humanity/Sonora) by rewriting that
> LayerNorm as a mathematically-identical channels-last norm, then re-exporting. Post-fix: `onnx2tf`
> op report 559/559 pass (was 361/457) and deterministic ONNX-vs-TFLite cosine = 1.0000 (was 0.045).
> The `.tflite` files here now carry the corrected weights under the same filenames.

---

## Provenance

**Convention (adopted 2026-07-13):** every artifact promotion into this registry records, in its
commit message, the training-repo state and run that produced it — `train-repo: <Sonora GitHub
SHA>` and `run: <MLflow run name>` (plus the export toolchain when relevant). Commits before this
date predate the convention.

**v1-ljspeech (backfilled, best-effort):**
* **Training:** MLflow run `lyrical-goat-689` on `ai-lab-0` (`sonora_training` container),
  2026-07-10 → 11; 260 epochs, checkpoint Epoch 199 selected at the validation-loss plateau.
  The exact training-repo SHA was not recorded at run time (pre-convention).
* **Export fix & re-export:** training repo commit
  [`a537e03`](https://github.com/Artificial-Humanity/Sonora/commit/a537e03) (channels-last
  encoder LayerNorm rewrite), re-exported 2026-07-12; fidelity gate
  `scripts/export_fidelity_referee.py` (ONNX↔TFLite cosine 1.0000, ASR WER 0.000).
* **Split lane:** same Epoch-199 weights, converted per-module with `litert-torch` via the
  [litert-samples](https://github.com/google-ai-edge/litert-samples) Matcha recipe.

---

## Training Environment & Hardware

The model was trained locally on the `ai-lab-0` workstation using a containerized AMD ROCm configuration:

* **Host CPU:** AMD Ryzen AI Max+ 395 (16 Cores, 32 Threads, Zen 5)
* **Host GPU:** AMD Radeon 8060S Graphics (RDNA 3.5, 40 Compute Units, Target: `gfx1151`)
* **Allocatable VRAM:** 112 GB Unified LPDDR5X (shared memory pool)
* **Operating System:** Ubuntu 26.04 LTS (Kernel `7.0.0-27-generic`)
* **Container Environment:** `rocm/pytorch:latest` running PyTorch 2.x + ROCm 7.2.4
* **Optimization:** 16-bit Automatic Mixed Precision (AMP), batch size 32, shared memory (`--ipc=host`, `--shm-size 16G`).

---

## Training History & Hyperparameters

* **Total Epochs Trained:** 260 (over 101,000 steps)
* **Selected Checkpoint:** Epoch 199 (step ~82,000) where validation loss converged and stabilized.
* **Loss Function:** Joint Flow-Matching loss, duration prediction loss, and monotonic alignment prior loss.
* **Troubleshooting Fixes Applied:**
  * **Audio Decoding:** Replaced `torchaudio.load()` with `soundfile.read()` in the data pipeline to bypass CUDA-exclusive shared library dependencies (`torchcodec` decoding failures) inside the ROCm container.
  * **Matplotlib Compatibility:** Patched validation figure logging to use `np.asarray(fig.canvas.buffer_rgba())[:, :, :3]`, avoiding the deprecated and removed `tostring_rgb()` method under Matplotlib 3.9+.

---

## Intended Uses & Limitations

### Intended Use
* **On-Device Narration:** Real-time text-to-speech synthesis directly on mobile devices (iOS/Android) and desktop clients without sending telemetry data to external cloud APIs.
* **Emotion & Directability Scaffolding:** Serves as the foundation for the Phase 1 directable voice actor, which will add Valence/Arousal/Tension (VAT) conditioning controls.

### Limitations
* **LJSpeech Speaker Profile:** The model is currently trained on a single female narrator and lacks voice variety or pitch modulation. Multi-speaker voice blending and casting grids are planned for the next training iteration (using Expresso and LibriTTS-R).
* **Phoneme Representation:** Requires an external G2P (Grapheme-to-Phoneme) converter to translate raw text into phoneme ID sequences before model execution. Project Prosodia handles this natively via its built-in CMUdict lexicon Rust crate (`crates/actor/src/g2p.rs`).

---

## How to Get Started

### Loading the Model in Project Prosodia (Rust)
The compiled model weights are loaded dynamically by the `StageCoordinator` through the TFLite runtime C-API. 

To use this model locally:
1. Download `checkpoint_epoch=199_e2e_float32.tflite` (the verified, `f32`-I/O artifact; the float16 build needs a delegate / engine `f16` support — see the format notes above) and the adjacent `v1-ljspeech/config.json` (the engine contract: locked 178-symbol vocab + `sample_rate` — the Prosodia engine reads it next to the model file).
2. Place both next to each other where your config points; name the model file whatever your role config declares (the Artificial-Humanity workspace uses `sonora.tflite`, resolved via the `actor` role in Prosodia's `prosodia_models.json`).
3. In the Artificial-Humanity workspace this is automated: the umbrella `bootstrap.sh` copies the blessed artifact + config into `Reference/models/`.
4. Build and run the harness:
   ```bash
   Prosodia/apps/tuner/build.sh
   ```

### Input/Output Tensor Node Contract
The model graph matches the following tensor specifications:
* **Inputs:**
  * `x`: token IDs representing phonemes (Shape: `[1, token_count]`, `i32` / `int64`).
  * `x_lengths`: length of the token sequence (Shape: `[1]`, `i32` / `int64`).
  * `scales`: scale parameters (Shape: `[2]`, `f32` containing `[temperature, length_scale]`).
* **Outputs:**
  * `wav`: Raw PCM audio waveform samples (Shape: `[1, audio_length]`, `f32` mono 22.05 kHz).
  * `wav_lengths`: The total count of generated audio samples (Shape: `[1]`, `i32` / `int64`).