--- title: Coding Agent Leaderboard emoji: 🥇 colorFrom: green colorTo: indigo sdk: gradio app_file: app.py pinned: true license: mit short_description: Compare coding agent models + harnesses sdk_version: 5.43.1 tags: - leaderboard --- # Coding Agent Leaderboard Compare coding-agent models and harnesses across benchmark performance, cost, latency, and token usage. ## Leaderboard views ### Efficiency The **Efficiency** tab compares benchmark score with resource use for one selected benchmark at a time. Keeping the view benchmark-specific avoids misleading comparisons when model and harness combinations have incomplete benchmark coverage. The generalized performance-vs-resource chart covers tokens, cost, and agent time in one benchmark-specific view. The scatter plot supports three resource metrics on the x-axis: - **Total tokens**: mean total token usage per task. - **Cost per task**: the repository's reported `mean_cost_usd_per_task` value, in USD. - **Agent time per task**: the repository's reported `mean_agent_time_seconds_per_task` value, in seconds. Only positive, available values for the selected resource are plotted. Missing, zero, and negative values are treated as unavailable rather than as perfect efficiency. The UI reports how many runs for the selected benchmark were excluded for the chosen resource metric. Scores are stored internally as fractions from 0 to 1 and displayed as percentages. Point labels are optional and coloring can be grouped by **Model** or **Harness**. Linear and logarithmic resource axes are available. #### Pareto frontier For all three resource metrics, lower resource use and higher score are better. A displayed run is Pareto-efficient when no other valid run on the selected benchmark: - uses an equal or lower amount of the selected resource, and - achieves an equal or higher score, with at least one strict improvement. Tied nondominated points are preserved. The dashed line connects the Pareto-efficient observations. #### Ranking table The ranking table remains benchmark-specific and begins with **Model**, **Harness**, and **Benchmark**. Displayed scores are rounded to one decimal place without reducing the precision used for calculations. **Tokens Per Solved Task** remains available in the table as a reference metric even though it is no longer an Efficiency scatter x-axis option. It is defined as `Total Tokens Per Task / Score`, where Score is the fractional value from 0 to 1. It is unavailable when total token data is missing/non-positive or when Score is zero or negative. #### Color palettes and themes The Efficiency chart uses the shared palette registry and supports Citrus, Okabe-Ito, High contrast, Rainbow, Grayscale, Viridis, Plasma, and Cividis palettes with light and dark chart backgrounds. ## Adding a New Leaderboard Entry Create a pull request adding a new entry to the `results/` folder. See [`results/qwen3-6-35b-nvfp4-claude-code.json`](./results/qwen3-6-35b-nvfp4-claude-code.json) for an example result. Do not change the result JSON schema for Efficiency analysis. The feature uses resource and performance metrics already present in the existing result model. ## Development 1. Install dependencies: ```sh pip install -r requirements.txt # or uv venv uv pip install -r requirements.txt ``` 2. Run the app: ```sh python app.py ``` 3. Run tests: ```sh pytest ``` ## Releasing Versions are managed with git tags. The `VERSION` file is the single source of truth for the version number (shown in the app header). Each release is tagged `v` on `main` by CI. Deployment targets: - **Production** (`red-hat-emerging-technologies/coding-agent-leaderboard`): only when a commit that changes `VERSION` lands on `main`. Other pushes to `main` do not deploy. To cut a release, bump the version in your PR: ```sh make bump VERSION=1.2.3 # writes VERSION and commits "Bump version to 1.2.3" git push # open / update your PR as usual ``` When the PR is merged into `main`, the **Release to HF Space (production)** workflow creates the `v1.2.3` tag on the merge commit and pushes it to the Space. If a tag for the current `VERSION` already exists, the workflow fails rather than moving the tag, so bump to a new number for every release. To redeploy an existing release (e.g. after a manual edit on the Space), run the workflow from the Actions tab and supply the tag. ## Manual validation After automated checks pass, launch the app from a clean process with `python app.py` and verify the Efficiency view in a real browser. Unit tests and figure-level smoke tests do not replace this browser validation. ### Efficiency behavior - Confirm there is no **All benchmarks** option and that a valid benchmark is selected by default. - Confirm **Color By** offers only **Model** and **Harness**. - Confirm the resource selector offers **Total tokens**, **Cost per task**, and **Agent time per task**. - Switch through all three resource metrics and verify axis labels and hover formatting. - Verify the Pareto frontier for all three metrics. - Verify linear and logarithmic scales where valid. - Verify point labels off and on. - Confirm the ranking table starts with **Model**, **Harness**, **Benchmark**, shows scores to one decimal place, and retains **Tokens Per Solved Task**. - Confirm the existing Leaderboard and Benchmark Runs tabs still work without new terminal tracebacks. ### Responsive layout verification - Restart the app from a clean launch. - Open **Efficiency** as the first non-default tab and confirm the plot is not squished. - Switch away from Efficiency and back several times. - Resize the browser narrower and wider. - Confirm the plot resizes correctly without requiring a control change. - Verify there is no legend overlap or clipping. - Verify the ranking table below does not force the plot into a narrow column. - Repeat the checks with point labels off and on, Pareto off and on, each resource metric, and light and dark plot backgrounds. - Also switch **Leaderboard → Efficiency** and **Benchmark Runs → Efficiency** to confirm hidden-tab initialization does not collapse the chart. ### PR completion report The final PR report should state the files changed, metric definitions, tests and results, data-quality limitations, and whether the app was manually verified. For the responsive-layout fix, it must additionally record: - the root cause of the initial squished-chart issue; - the exact responsive-layout fix used; and - whether the behavior was manually verified from a fresh app launch.