RyanL22/pi05-rby1-wujihand2-egomimic-baseline-20k

pi0.5 baseline -- egomimic visual alignment (RB-Y1 + Wujihand2), step 20k

LeRobot 0.6.1 pi05_base fine-tuned as a human-video baseline for RB-Y1 with Wujihand2 hands. One of two visual-alignment baselines trained on identical data, labels and hyper-parameters; only the image edit differs, so the two are directly comparable.

EgoMimic style: the SAM3 arm mask is filled black (3 px dilation) and a red line is drawn over it -- line_on_hand inside the mask bounding box for human frames, and the FK palm-to-elbow segment for robot teleop frames (20 px at 512 width).

Data

source	episodes	note
RB-Y1 teleop v1	209	4 categories (ball, bottle, small box, doll), real robot
RoboTryOn-human-2	451	11 categories, human video re-shot to match the RB-Y1 ZED view

660 episodes, 170,971 frames, 30 fps, stereo 288x512 (both ZED eyes). 15 cells (4 robot + 11 human), share proportional to sqrt(frames).

Human labels come from HaWoR stereo triangulation retargeted to RB-Y1: 7-DoF damped-least-squares arm IK to the wrist pose plus per-finger IK for Wujihand2. The action leads the image by 7 (arm) / 4 (hand) frames, matching the measured teleop lead; the render is drawn at the label pose, not the action.

Policy

• state / action: 54 dims = observation.joint_position[10:64] (right arm 7, right hand 20, left arm 7, left hand 20). Wheels, torso and head are constant in this data and excluded.
• base expanded from 32 to 54 dims, tokenizer max length 320.
• 20,000 steps, global batch 64 (2 GPUs x 32) on H100, bfloat16, vision encoder unfrozen, stereo-consistent photometric augmentation, no mirror augmentation. Final loss 0.017.
• camera mapping at training and inference: left eye -> observation.images.base_0_rgb, right eye -> observation.images.left_wrist_0_rgb.

Known limitation

12 air-fryer episodes have IK wrist residuals of 2-5 cm because the left hand opening the door is outside RB-Y1's reach. They were kept, as in the OpenArm baselines. Masquerade stays image-label consistent there since it renders the label pose; the EgoMimic red line follows the human hand and so disagrees with the label by a few cm.