README.md

metadata

title: Real World VoiceEQ Benchmark
emoji: 🎙️
colorFrom: indigo
colorTo: green
sdk: gradio
sdk_version: 6.14.0
app_file: app.py
pinned: false
license: apache-2.0

Real World VoiceEQ Benchmark

One leaderboard, six tabs: Text-to-Speech, Voice Controllability, Speech-to-Speech, Speech Understanding, Speech Recognition, and SLM Judge (a meta-leaderboard: how well each SLM judge tracks human ratings). Each tab is a heatmap ranking of providers across that modality's factors. Rows are ranked by the first factor column by default; clicking any column header re-sorts and re-ranks, and rows with no value in the active column sink to the bottom, unranked.

Data

The tables are loaded at runtime from a Hugging Face dataset holding all six board JSONs (tts_leaderboard.json, sts_leaderboard.json, voice_creation_leaderboard.json, stt_leaderboard.json, asr_leaderboard.json, slm_judge_leaderboard.json). Configure it with two Space settings:

• Variable LEADERBOARD_DATASET → e.g. HumeAI/hume-speech-leaderboard-data
• Secret HF_TOKEN → a read token (only needed if the dataset is private)

For local development, the app falls back to the files in ./data/.

Local preview

pip install -r requirements.txt
python app.py

Data schema

Each board JSON is self-describing: a list of board objects — an Overall board plus one per factor. The app pivots the factor boards into one wide table (one heatmapped column per factor). The Overall board supplies per-provider license and sizeB; its composite score and the coverage count stay in the data but are not displayed. Single-factor modalities (Voice Controllability) ship just their factor board, no Overall board. license is shown in its own column; null metrics render as ·. Rater-scored tabs heatmap on the absolute 1–5 scale; boards declaring a heatmap mode use their own scale instead — Speech Recognition's WER columns share fixed absolute anchors ({"mode": "absolute", "stops": [...]}), with each column's direction honoured so lower-is-better metrics still shade best-as-green. Full key-by-key breakdown is in data/README.md.