---
title: RoboEval Challenge CoRL 2026
emoji: 🤖
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
hf_oauth: true
pinned: true
short_description: Submissions and leaderboard for the RoboEval bimanual track
---

# RoboEval challenge portal

Submission portal and leaderboard for the CoRL 2026 RoboEval Diagnostic
Bimanual Challenge. Participants sign in with Hugging Face, submit the archive
built by `python -m roboeval.challenge.package`, and get the validation report
back in a few minutes.

## What happens to a submission

1. **Gate:** submissions open, before the deadline, a 64-hex SHA-256 given.
2. **Fetch:** the archive is downloaded from its https link (or taken from the
   upload) up to `MAX_ARCHIVE_GB`, and must match the given SHA-256.
3. **Unpack safely:** only regular files under `submission/`; no links,
   absolute or `..` paths, or archives that expand past the limit. Checkpoint
   files are hashed but not stored.
4. **Manifest:** every file must match `MANIFEST.sha256`, with nothing extra.
5. **Identity:** one Hugging Face account per team and one team per account
   (the first accepted submission binds them). Then the team's limits apply.
6. **Validation:** `roboeval.challenge.validate` with `REPLAY_EPISODES` replays,
   on the same release participants use, in the same pinned numeric
   environment. No participant code is executed.
7. **Decision:** all checks pass → **accepted** (on the leaderboard). Only the
   replay fails → **needs_review** (or rejected, per `REPLAY_MISMATCH`).
   Anything else → **rejected**; rejected bundles do not count toward limits.

Everything is stored in a private dataset repo: per submission, `record.json`,
`results.json`, `metadata.yaml`, `validation.json`, and the archive itself when
it is under `STORE_ARCHIVE_MAX_GB` (for audits and held-out re-runs).

The leaderboard shows each team's most recent accepted submission, ranked by
the RoboEval Diagnostic Score (`roboeval/challenge/scoring.py`), then Success,
Task Progression, the safety score, and submission time. It has per-task and
per-metric-family views and filters by observation and action mode.
Organizers' own entries are labelled not prize-eligible.

## Configuration

Space **secrets**: `HF_TOKEN`, a token with write access to the store dataset.

Space **variables**:

| variable | default | meaning |
|---|---|---|
| `PORTAL_STORE` | `local:./portal_data` | `hf:<namespace>/<dataset>` in production (created private if missing) |
| `SUBMISSIONS_OPEN` | `true` | master switch |
| `SUBMISSION_DEADLINE` | unset | ISO time, e.g. `2026-11-01T23:59:00-07:00` |
| `SUBMISSION_LIMIT_PER_DAY` | `2` | per team per UTC day; `none` for unlimited |
| `SUBMISSION_LIMIT_TOTAL` | unset | per team over the challenge |
| `MAX_ARCHIVE_GB` | `5` | download and upload cap |
| `STORE_ARCHIVE_MAX_GB` | `2` | keep a copy of archives up to this size |
| `REPLAY_EPISODES` | `20` | episodes replayed per submission |
| `REPLAY_MISMATCH` | `review` | `review` or `reject` |
| `ORGANIZERS` | unset | comma-separated HF usernames: Organizers tab, entries marked not prize-eligible |

The Docker build arguments `ROBOEVAL_REPO` and `ROBOEVAL_REF` choose the
RoboEval release the validator runs; keep `ROBOEVAL_REF` on the challenge tag.

## Deploying

The Space repo holds the contents of this `portal/` directory.

```bash
hf repo create <namespace>/roboeval-challenge --repo-type space --space-sdk docker
hf upload <namespace>/roboeval-challenge portal/ . --repo-type space
```

Then set the secret and variables above in the Space settings. CPU basic
hardware is enough: a submission validates in a few minutes.

## Organizer tasks

- **Review queue:** the Organizers tab lists every submission. Approve a
  `needs_review` entry by setting it to `accepted`, or hide a disqualified
  one with `hidden`; the note is kept in its record.
- **Held-out re-runs:** download the archive from the store and run
  `python -m roboeval.challenge.evaluate --submission <dir> --seeds-file heldout.json --output-dir <out>`
  (the checkpoint comes from the participant's `checkpoint_url` or a direct request).
- **Website:** `python export_leaderboard.py --out <site>/static/data` writes
  `challenge_leaderboard_*.json` in the site's leaderboard format.

## Local development

```bash
pip install -e <RoboEval checkout> -r requirements.txt
OPENBLAS_CORETYPE=Haswell OPENBLAS_NUM_THREADS=1 python app.py   # store defaults to ./portal_data
PORTAL_TEST_BUNDLE=<archive.tar.gz> pytest test_portal.py
```