File size: 2,064 Bytes
90d34be | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | # Evaluation metrics
These figures are a frozen, source-derived comparison of the merged `step1500` checkpoint using reference `r2`, seed `42`, `cfg=2.0`, and `16` timesteps. The two development cases contain 151 and 149 words.
| Case | Method | Audio duration | Measured time | Ratio / RTF | First chunk | Peak allocated / reserved |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| library (151 words) | whole call | 59.04 s | 21.778395 s | RTF 0.368875 | 0.582673 s | 5638.327 / 5978.0 MiB |
| library (151 words) | sentence reset, 12 calls | 68.16 s (sum) | 24.521308 s (sum) | 0.359761 aggregate | 0.062772 s | 5750.052 / 6114.0 MiB |
| platform (149 words) | whole call | 59.36 s | 21.451940 s | RTF 0.361387 | 0.067999 s | 5749.552 / 6114.0 MiB |
| platform (149 words) | sentence reset, 13 calls | 66.72 s (sum) | 24.042400 s (sum) | 0.360348 aggregate | 0.067221 s | 5750.052 / 6114.0 MiB |
For sentence reset, the aggregate ratio is `sum(elapsed_s) / sum(duration_s)` over calls. It excludes file I/O, model loading, inter-call gaps, and assembly, so it is not end-to-end service latency. Whole-call values use the recorded row RTF. The library whole-call row was the first model call in the process; the remaining rows were warm. First-chunk figures exclude model load and should not be read as a matched warmup-latency comparison.
ASR word-error counts were 1/151 (whole) and 0/151 (reset) for the library case; both methods were 2/149 for the platform case. A single user's two trials rated reset naturalness/similarity `5/5` and `4/5` versus whole-call `1/1` in both cases, preferring reset in both. The second reset artifact field was unanswered (`[]`); it is not treated as evidence of no artifact. These are one-user ratings, not a MOS or universal checkpoint claim.
The exact manually authored sentences are in `../examples/library.sentences.txt` and `../examples/platform.sentences.txt`. Model, source, protocol, reference, and scorer hashes are recorded in `metrics.json`. No reference WAV or private participant identifier is included.
|