---
title: Real World VoiceEQ Benchmark
emoji: 🎙️
colorFrom: indigo
colorTo: green
sdk: gradio
sdk_version: 6.14.0
app_file: app.py
pinned: false
license: apache-2.0
---

# Real World VoiceEQ Benchmark

One leaderboard: an **About** tab (overview from `about.json` + one clickable
card per panel), then **Text-to-Speech**, **Speech-to-Speech**,
**Speech Understanding**, and **ASR**. Each leaderboard tab is a heatmap
ranking of providers across that modality's factors. (The SLM Judge
meta-leaderboard lives in its own Space, built from the same dataset.) Rows are
ranked by the first factor column by default; clicking any column header re-sorts
and re-ranks, and rows with no value in the active column sink to the bottom,
unranked.

## Data

The tables are loaded at runtime from a Hugging Face **dataset** holding all six
board JSONs (`tts_leaderboard.json`, `sts_leaderboard.json`,
`voice_creation_leaderboard.json`, `stt_leaderboard.json`,
`asr_leaderboard.json`, `slm_judge_leaderboard.json`). Configure it with two
Space settings:

- Variable `LEADERBOARD_DATASET` → e.g. `HumeAI/hume-speech-leaderboard-data`
- Secret `HF_TOKEN` → a read token (only needed if the dataset is private)

For local development, the app falls back to the files in `./data/`.

## Local preview

```bash
pip install -r requirements.txt
python app.py
```

## Data schema

Each board JSON is self-describing: a list of board objects — an **Overall** board
plus one **per factor**. The app pivots the factor boards into one wide table (one
heatmapped column per factor). The Overall board supplies per-provider `license`
and `sizeB`; its composite score and the `coverage` count stay in the data but are
not displayed. Single-factor files ship just their factor
board, no Overall board. `license` is shown in its own column; null metrics render
as `·`. Rater-scored tabs heatmap on the absolute 1–5 scale; boards declaring a
`heatmap` mode use their own scale instead — the ASR tab's WER columns
share fixed absolute anchors (`{"mode": "absolute", "stops": [...]}`), with each
column's `direction` honoured so lower-is-better metrics still shade
best-as-green. Full key-by-key breakdown is in
[`data/README.md`](data/README.md).