Evaluation metrics
These figures are a frozen, source-derived comparison of the merged step1500 checkpoint using reference r2, seed 42, cfg=2.0, and 16 timesteps. The two development cases contain 151 and 149 words.
| Case | Method | Audio duration | Measured time | Ratio / RTF | First chunk | Peak allocated / reserved |
|---|---|---|---|---|---|---|
| library (151 words) | whole call | 59.04 s | 21.778395 s | RTF 0.368875 | 0.582673 s | 5638.327 / 5978.0 MiB |
| library (151 words) | sentence reset, 12 calls | 68.16 s (sum) | 24.521308 s (sum) | 0.359761 aggregate | 0.062772 s | 5750.052 / 6114.0 MiB |
| platform (149 words) | whole call | 59.36 s | 21.451940 s | RTF 0.361387 | 0.067999 s | 5749.552 / 6114.0 MiB |
| platform (149 words) | sentence reset, 13 calls | 66.72 s (sum) | 24.042400 s (sum) | 0.360348 aggregate | 0.067221 s | 5750.052 / 6114.0 MiB |
For sentence reset, the aggregate ratio is sum(elapsed_s) / sum(duration_s) over calls. It excludes file I/O, model loading, inter-call gaps, and assembly, so it is not end-to-end service latency. Whole-call values use the recorded row RTF. The library whole-call row was the first model call in the process; the remaining rows were warm. First-chunk figures exclude model load and should not be read as a matched warmup-latency comparison.
ASR word-error counts were 1/151 (whole) and 0/151 (reset) for the library case; both methods were 2/149 for the platform case. A single user's two trials rated reset naturalness/similarity 5/5 and 4/5 versus whole-call 1/1 in both cases, preferring reset in both. The second reset artifact field was unanswered ([]); it is not treated as evidence of no artifact. These are one-user ratings, not a MOS or universal checkpoint claim.
The exact manually authored sentences are in ../examples/library.sentences.txt and ../examples/platform.sentences.txt. Model, source, protocol, reference, and scorer hashes are recorded in metrics.json. No reference WAV or private participant identifier is included.