examples/LIBERO/smoke_tests/README.md

LIBERO simulation-only smoke tests for nvidia/GR00T-N1.7-LIBERO

These scripts deploy the libero_10 checkpoint of nvidia/GR00T-N1.7-LIBERO and run 10 short, simulation-only LIBERO rollouts to verify the deploy / eval plumbing:

1. the model checkpoint is present and loads,
2. the GR00T inference server starts,
3. the LIBERO rollout client connects to it,
4. short rollouts run with no physical robot hardware (LIBERO sim only),
5. a video / rendered frames are saved per scenario for human review,
6. basic rollout metadata + recorded action chunks are saved,
7. activation hooks can be added later — see ACTIVATION_HOOK_NOTES.md.

This is not a benchmark. max_episode_steps is deliberately tiny, so the reported success rates are meaningless — only the plumbing matters here. The scripts do not rewrite the official evaluation path: each scenario runs through gr00t.eval.rollout_policy.run_rollout_gymnasium_policy via the thin _libero_rollout_worker.py, which only adds action recording and two optional, simulation-only perturbations (Gaussian pixel noise; action-chunk repeat).

Files

File	Purpose
scenarios_10.yaml	the 10-scenario manifest (8 normal, 2 abnormal_probe)
run_10_smoke_tests.py	runner: validates paths/server, runs all scenarios, writes summaries
_libero_rollout_worker.py	runs one rollout against the server (called by the runner; runs in the LIBERO venv)
review_smoke_tests.py	lists scenarios + video paths from a run dir, optionally plays videos
ACTIVATION_HOOK_NOTES.md	where to insert future SAE activation capture

Tests live at tests/test_libero_smoke_tests.py.

One-time setup

# 1. Install deps (project venv) — uses uv
uv sync          # or: uv pip install -e .   (see note below)

# 2. LIBERO simulation env (separate venv; only needed once)
sudo apt update && sudo apt install libegl1-mesa-dev libglu1-mesa
bash gr00t/eval/sim/LIBERO/setup_libero.sh

# 3. Download the libero_10 checkpoint
#    (HuggingFace does not support nested repo paths directly)
uv run hf download nvidia/GR00T-N1.7-LIBERO \
  --include "libero_10/config.json" \
            "libero_10/embodiment_id.json" \
            "libero_10/model-*.safetensors" \
            "libero_10/model.safetensors.index.json" \
            "libero_10/processor_config.json" \
            "libero_10/statistics.json" \
  --local-dir checkpoints/GR00T-N1.7-LIBERO
# If the glob skips config.json, fetch it explicitly:
uv run hf download nvidia/GR00T-N1.7-LIBERO libero_10/config.json \
  --local-dir checkpoints/GR00T-N1.7-LIBERO

Note on uv sync: the project pyproject.toml pins GPU-deployment-only packages (flash-attn, deepspeed, tensorrt-*) and the lockfile references aarch64 wheels that are stored via git-LFS. On a plain x86 box without git-LFS / CUDA toolkit, uv sync can fail on those. They are not needed for the smoke test (the model falls back to sdpa attention if flash_attn is missing). A working minimal install is:

uv venv .venv --python 3.10
uv pip install --python .venv/bin/python -e . --no-deps
uv pip install --python .venv/bin/python \
  torch==2.7.1 torchvision==0.22.1 transformers==4.57.3 numpy==1.26.4 \
  albumentations==1.4.18 av==16.1.0 diffusers==0.35.1 dm-tree lmdb==1.7.5 \
  msgpack==1.1.0 msgpack-numpy==0.4.8 pandas==2.2.3 peft==0.17.1 termcolor==3.2.0 \
  tyro==0.9.17 click==8.1.8 datasets==3.6.0 cryptography einops==0.8.1 \
  gitpython==3.1.46 jsonlines==4.0.0 gymnasium==1.2.2 matplotlib==3.10.1 \
  omegaconf==2.3.0 scipy==1.15.3 torchcodec==0.4.0 wandb==0.23.0 pyzmq==27.0.1 \
  "huggingface-hub[cli]" "opencv-python-headless>=4.5,<4.13" safetensors accelerate \
  sentencepiece protobuf pyyaml tqdm

Running the smoke tests

Terminal 1 — start the GR00T inference server:

uv run python gr00t/eval/run_gr00t_server.py \
    --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
    --embodiment-tag LIBERO_PANDA \
    --use-sim-policy-wrapper

(Server defaults: --host 0.0.0.0 --port 5555. This is exactly the command documented in examples/LIBERO/README.md.)

Gated backbone: GR00T-N1.7 uses nvidia/Cosmos-Reason2-2B (a Qwen3-VL model) as its VLM backbone, and that repo is gated on HuggingFace. Starting the server pulls the base repo's config/processor, so it fails without HF auth. One-time fix:

# 1. request access (one click): https://huggingface.co/nvidia/Cosmos-Reason2-2B
# 2. authenticate:
export HF_TOKEN=hf_xxx          # or: uv run hf auth login

The run_10_smoke_tests.py runner detects this failure mode (when launched with --start-server) and prints the same hint.

Terminal 2 — run the 10 smoke tests:

uv run python examples/LIBERO/smoke_tests/run_10_smoke_tests.py \
    --model-path checkpoints/GR00T-N1.7-LIBERO/libero_10 \
    --host 127.0.0.1 --port 5555 \
    --manifest examples/LIBERO/smoke_tests/scenarios_10.yaml \
    --output-dir outputs/libero_smoke_tests \
    --max-episode-steps 50 \
    --save-video --render

Other flags:

• --start-server — launch (and later kill) the server from the runner itself, so you only need one terminal.
• --dry-run — validate the manifest, model path and (optionally) the server, write metadata.json per scenario, but run no simulations.
• --resume + --run-dir <output-dir>/<timestamp> — re-enter an existing run dir and skip scenarios that already produced a rollout_summary.json.
• --libero-python <path> — override the LIBERO venv python (default: gr00t/eval/sim/LIBERO/libero_uv/.venv/bin/python, falls back to the current interpreter with a warning).

If the model path is missing, or the server is unreachable, or the LIBERO sim env is not importable, the runner stops with an actionable error — it does not fake a successful rollout.

Outputs

outputs/libero_smoke_tests/<timestamp>/
  summary.json
  summary.md
  server.log                       # only if --start-server
  <scenario_id>/
    metadata.json                  # the manifest entry + resolved params
    rollout_summary.json           # env_name, seed, steps, success, reward, ...
    actions.npy                    # recorded action chunks (if the rollout ran)
    video.mp4                      # rendered rollout (if recording succeeded)
    frames/                        # decoded PNG frames (only with --render)
    stdout.log / stderr.log        # the worker subprocess logs

At the end the runner prints a table:

scenario_id | label | seed | rollout_started | actions_produced | video_saved | success | output_dir | error_if_any

and writes the same to summary.json / summary.md. Review videos with:

python examples/LIBERO/smoke_tests/review_smoke_tests.py --run-dir outputs/libero_smoke_tests/<timestamp> --open