hud-evals
/

pi05-AssembleBench-cgdagger-r3

@@ -1,72 +1,73 @@
----
-license: apache-2.0
-library_name: lerobot
-pipeline_tag: robotics
-tags:
-  - robotics
-  - vla
-  - pi0.5
-  - assembly
-  - isaac-sim
-base_model: DAVIAN-Robotics/pi05_droid_jointpos
-datasets:
-  - hud-evals/AssembleBench
----
-# pi0.5 AssembleBench - CG-DAgger round 3 (final)
-The final checkpoint from the writeup: the BC baseline plus three rounds of code-gated DAgger. This is what `examples/run_eval.py` loads by default.
-pi0.5 finetuned for [AssembleBench](https://github.com/hud-evals/assemble-bench), a
-14-task contact-rich robot assembly benchmark on the NIST ATB-1 taskboard: insert round
-and square pegs, mesh gears, thread nuts. Built on Isaac Lab Arena with the DROID robot
-platform (Franka Panda + Robotiq 2F-85).
-Read the writeup: [Benchmarking Robot Models on Contact-Rich Assembly](https://www.hud.ai/blog/assemble-benchmark).
-## Usage
-```bash
-git clone https://github.com/hud-evals/assemble-bench.git
-cd assemble-bench
-pip install -r examples/requirements.txt
-# Serve the simulator (see the repo README for the Isaac Sim install)
-OMNI_KIT_ACCEPT_EULA=YES python -m hud.environment.server env.py --port 8765
-# Evaluate this checkpoint
-python examples/run_eval.py --task peg_round_16mm --checkpoint hud-evals/pi05-AssembleBench-cgdagger-r3
-```
-The DROID input/output convention is not baked into the checkpoint. `examples/pi05_agent.py`
-carries the adapter: front and wrist RGB resize-padded to 224, an 8-D state of 7 joint
-positions plus gripper, and 8-D absolute joint-target actions at 15 Hz with a binarized
-gripper.
-## Training
-Finetuned from [`DAVIAN-Robotics/pi05_droid_jointpos`](https://huggingface.co/DAVIAN-Robotics/pi05_droid_jointpos)
-(full finetune, VLM included) on
-[`hud-evals/AssembleBench`](https://huggingface.co/datasets/hud-evals/AssembleBench),
-1355 synthetic demonstrations generated by scripted experts reading privileged simulator
-state, with no teleoperation.
-Code-gated DAgger (CG-DAgger): the policy drives, code detects failure and hands control to a scripted expert that demonstrates a recovery, and the policy is retrained on those intervention segments. It is HG-DAgger with the human replaced by a failure gate. Corrections were collected only on `peg_round_8mm`, so the other round-peg sizes measure whether the recovery transfers.
-Round 3 targeted **insertion failure** from releasing the grasp mid-insertion. Recovery: insert without ever opening the gripper. Later rounds improved the looser clearances (12 mm and 16 mm); 4 mm stayed flat.
-## Limitations
-Task-specific: this is not a generalist checkpoint, and it is trained purely in simulation.
-Success on these tasks is sparse - a handful of wins out of thirty on the harder families -
-so the writeup reports dense reward alongside binary success. Square pegs and small gears
-remain close to zero.
-## Related checkpoints
-| Checkpoint | What it is |
-|---|---|
-| [`pi05-AssembleBench-12k`](https://huggingface.co/hud-evals/pi05-AssembleBench-12k) | behavior-cloning baseline, all 14 tasks |
-| [`pi05-AssembleBench-cgdagger-r1`](https://huggingface.co/hud-evals/pi05-AssembleBench-cgdagger-r1) | + code-gated DAgger round 1 |
-| [`pi05-AssembleBench-cgdagger-r2`](https://huggingface.co/hud-evals/pi05-AssembleBench-cgdagger-r2) | + round 2 |
-| [`pi05-AssembleBench-cgdagger-r3`](https://huggingface.co/hud-evals/pi05-AssembleBench-cgdagger-r3) | + round 3, the final checkpoint |

+---
+license: apache-2.0
+library_name: lerobot
+pipeline_tag: robotics
+tags:
+  - robotics
+  - vla
+  - pi0.5
+  - assembly
+  - isaac-sim
+base_model: DAVIAN-Robotics/pi05_droid_jointpos
+datasets:
+  - hud-evals/AssembleBench
+---
+# pi0.5 AssembleBench - CG-DAgger round 3 (final)
+The final checkpoint from the writeup: the BC baseline plus three rounds of code-gated DAgger. This is what `examples/run_eval.py` loads by default.
+pi0.5 finetuned for [AssembleBench](https://github.com/hud-evals/assemble-bench), a
+14-task contact-rich robot assembly benchmark on the NIST ATB-1 taskboard: insert round
+and square pegs, mesh gears, thread nuts. Built on Isaac Lab Arena with the DROID robot
+platform (Franka Panda + Robotiq 2F-85).
+Read the writeup: [Benchmarking Robot Models on Contact-Rich Assembly](https://www.hud.ai/blog/assemble-benchmark).
+## Usage
+```bash
+git clone https://github.com/hud-evals/assemble-bench.git
+cd assemble-bench
+pip install -r examples/requirements.txt
+# Serve the simulator (see the repo README for the Isaac Sim install)
+OMNI_KIT_ACCEPT_EULA=YES python -m hud.environment.server env.py --port 8765
+# Evaluate this checkpoint
+python examples/run_eval.py --task peg_round_16mm --checkpoint hud-evals/pi05-AssembleBench-cgdagger-r3
+```
+The DROID input/output convention is not baked into the checkpoint. `examples/pi05_agent.py`
+carries the adapter: front and wrist RGB resize-padded to 224, an 8-D state of 7 joint
+positions plus gripper, and 8-D absolute joint-target actions at 15 Hz with a binarized
+gripper.
+## Training
+Finetuned from [`DAVIAN-Robotics/pi05_droid_jointpos`](https://huggingface.co/DAVIAN-Robotics/pi05_droid_jointpos)
+(full finetune, VLM included) on
+[`hud-evals/AssembleBench`](https://huggingface.co/datasets/hud-evals/AssembleBench),
+1355 synthetic demonstrations generated by scripted experts reading privileged simulator
+state, with no teleoperation.
+Code-gated DAgger (CG-DAgger): the policy drives, code detects failure and hands control to a scripted expert that demonstrates a recovery, and the policy is retrained on those intervention segments. It is HG-DAgger with the human replaced by a failure gate. Corrections were collected only on `peg_round_8mm`, so the other round-peg sizes measure whether the recovery transfers.
+Three rounds were collected. Round 1 targeted grasp failure (grip lower and reattempt),
+round 2 off-center peg placement (re-center before inserting), and round 3 insertion
+failure from releasing the grasp mid-insertion (insert without ever opening the gripper).
+Later rounds improved the looser clearances (12 mm and 16 mm); 4 mm stayed flat.
+## Limitations
+Task-specific: this is not a generalist checkpoint, and it is trained purely in simulation.
+Success on these tasks is sparse - a handful of wins out of thirty on the harder families -
+so the writeup reports dense reward alongside binary success. Square pegs and small gears
+remain close to zero.
+## Related checkpoints
+| Checkpoint | What it is |
+|---|---|
+| [`pi05-AssembleBench-12k`](https://huggingface.co/hud-evals/pi05-AssembleBench-12k) | behavior-cloning baseline, all 14 tasks |
+| [`pi05-AssembleBench-cgdagger-r3`](https://huggingface.co/hud-evals/pi05-AssembleBench-cgdagger-r3) | BC + 3 rounds of code-gated DAgger, the final checkpoint |