# LIBERO simulation-only smoke tests for `nvidia/GR00T-N1.7-LIBERO`

These scripts deploy the `libero_10` checkpoint of
[`nvidia/GR00T-N1.7-LIBERO`](https://huggingface.co/nvidia/GR00T-N1.7-LIBERO)
and run **10 short, simulation-only LIBERO rollouts** to verify the deploy /
eval plumbing:

1. the model checkpoint is present and loads,
2. the GR00T inference server starts,
3. the LIBERO rollout client connects to it,
4. short rollouts run with **no physical robot hardware** (LIBERO sim only),
5. a video / rendered frames are saved per scenario for human review,
6. basic rollout metadata + recorded action chunks are saved,
7. activation hooks can be added later — see [`ACTIVATION_HOOK_NOTES.md`](ACTIVATION_HOOK_NOTES.md).

This is **not** a benchmark. `max_episode_steps` is deliberately tiny, so the
reported success rates are meaningless — only the plumbing matters here. The
scripts do **not** rewrite the official evaluation path: each scenario runs
through `gr00t.eval.rollout_policy.run_rollout_gymnasium_policy` via the thin
`_libero_rollout_worker.py`, which only adds action recording and two optional,
simulation-only perturbations (Gaussian pixel noise; action-chunk repeat).

## Files

| File | Purpose |
|---|---|
| `scenarios_10.yaml` | the 10-scenario manifest (8 `normal`, 2 `abnormal_probe`) |
| `run_10_smoke_tests.py` | runner: validates paths/server, runs all scenarios, writes summaries |
| `_libero_rollout_worker.py` | runs **one** rollout against the server (called by the runner; runs in the LIBERO venv) |
| `review_smoke_tests.py` | lists scenarios + video paths from a run dir, optionally plays videos |
| `ACTIVATION_HOOK_NOTES.md` | where to insert future SAE activation capture |

Tests live at `tests/test_libero_smoke_tests.py`.

## One-time setup

```bash
# 1. Install deps (project venv) — uses uv
uv sync          # or: uv pip install -e .   (see note below)

# 2. LIBERO simulation env (separate venv; only needed once)
sudo apt update && sudo apt install libegl1-mesa-dev libglu1-mesa
bash gr00t/eval/sim/LIBERO/setup_libero.sh

# 3. Download the libero_10 checkpoint
#    (HuggingFace does not support nested repo paths directly)
uv run hf download nvidia/GR00T-N1.7-LIBERO \
  --include "libero_10/config.json" \
            "libero_10/embodiment_id.json" \
            "libero_10/model-*.safetensors" \
            "libero_10/model.safetensors.index.json" \
            "libero_10/processor_config.json" \
            "libero_10/statistics.json" \
  --local-dir checkpoints/GR00T-N1.7-LIBERO
# If the glob skips config.json, fetch it explicitly:
uv run hf download nvidia/GR00T-N1.7-LIBERO libero_10/config.json \
  --local-dir checkpoints/GR00T-N1.7-LIBERO
```

> **Note on `uv sync`:** the project `pyproject.toml` pins GPU-deployment-only
> packages (`flash-attn`, `deepspeed`, `tensorrt-*`) and the lockfile references
> aarch64 wheels that are stored via git-LFS. On a plain x86 box without git-LFS
> / CUDA toolkit, `uv sync` can fail on those. They are **not needed for the
> smoke test** (the model falls back to `sdpa` attention if `flash_attn` is
> missing). A working minimal install is:
> ```bash
> uv venv .venv --python 3.10
> uv pip install --python .venv/bin/python -e . --no-deps
> uv pip install --python .venv/bin/python \
>   torch==2.7.1 torchvision==0.22.1 transformers==4.57.3 numpy==1.26.4 \
>   albumentations==1.4.18 av==16.1.0 diffusers==0.35.1 dm-tree lmdb==1.7.5 \
>   msgpack==1.1.0 msgpack-numpy==0.4.8 pandas==2.2.3 peft==0.17.1 termcolor==3.2.0 \
>   tyro==0.9.17 click==8.1.8 datasets==3.6.0 cryptography einops==0.8.1 \
>   gitpython==3.1.46 jsonlines==4.0.0 gymnasium==1.2.2 matplotlib==3.10.1 \
>   omegaconf==2.3.0 scipy==1.15.3 torchcodec==0.4.0 wandb==0.23.0 pyzmq==27.0.1 \
>   "huggingface-hub[cli]" "opencv-python-headless>=4.5,<4.13" safetensors accelerate \
>   sentencepiece protobuf pyyaml tqdm
> ```

## Running the smoke tests

**Terminal 1 — start the GR00T inference server:**

```bash
uv run python gr00t/eval/run_gr00t_server.py \
    --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
    --embodiment-tag LIBERO_PANDA \
    --use-sim-policy-wrapper
```

(Server defaults: `--host 0.0.0.0 --port 5555`. This is exactly the command
documented in `examples/LIBERO/README.md`.)

> **Gated backbone:** GR00T-N1.7 uses `nvidia/Cosmos-Reason2-2B` (a Qwen3-VL
> model) as its VLM backbone, and that repo is **gated** on HuggingFace.
> Starting the server pulls the base repo's config/processor, so it fails
> without HF auth. One-time fix:
> ```bash
> # 1. request access (one click): https://huggingface.co/nvidia/Cosmos-Reason2-2B
> # 2. authenticate:
> export HF_TOKEN=hf_xxx          # or: uv run hf auth login
> ```
> The `run_10_smoke_tests.py` runner detects this failure mode (when launched
> with `--start-server`) and prints the same hint.

**Terminal 2 — run the 10 smoke tests:**

```bash
uv run python examples/LIBERO/smoke_tests/run_10_smoke_tests.py \
    --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
    --host 127.0.0.1 --port 5555 \
    --manifest examples/LIBERO/smoke_tests/scenarios_10.yaml \
    --output-dir outputs/libero_smoke_tests \
    --max-episode-steps 50 \
    --save-video --render
```

Other flags:

- `--start-server` — launch (and later kill) the server from the runner itself,
  so you only need one terminal.
- `--dry-run` — validate the manifest, model path and (optionally) the server,
  write `metadata.json` per scenario, but run no simulations.
- `--resume` + `--run-dir <output-dir>/<timestamp>` — re-enter an existing run
  dir and skip scenarios that already produced a `rollout_summary.json`.
- `--libero-python <path>` — override the LIBERO venv python (default:
  `gr00t/eval/sim/LIBERO/libero_uv/.venv/bin/python`, falls back to the current
  interpreter with a warning).

If the model path is missing, or the server is unreachable, or the LIBERO sim
env is not importable, the runner stops with an actionable error — it does
**not** fake a successful rollout.

## Outputs

```
outputs/libero_smoke_tests/<timestamp>/
  summary.json
  summary.md
  server.log                       # only if --start-server
  <scenario_id>/
    metadata.json                  # the manifest entry + resolved params
    rollout_summary.json           # env_name, seed, steps, success, reward, ...
    actions.npy                    # recorded action chunks (if the rollout ran)
    video.mp4                      # rendered rollout (if recording succeeded)
    frames/                        # decoded PNG frames (only with --render)
    stdout.log / stderr.log        # the worker subprocess logs
```

At the end the runner prints a table:

```
scenario_id | label | seed | rollout_started | actions_produced | video_saved | success | output_dir | error_if_any
```

and writes the same to `summary.json` / `summary.md`. Review videos with:

```bash
python examples/LIBERO/smoke_tests/review_smoke_tests.py --run-dir outputs/libero_smoke_tests/<timestamp> --open
```