RyanL22/pi05-rby1-wujihand2-masquerade-baseline-20k

pi0.5 baseline -- masquerade visual alignment (RB-Y1 + Wujihand2), step 20k

LeRobot 0.6.1 pi05_base fine-tuned as a human-video baseline for RB-Y1 with Wujihand2 hands. One of two visual-alignment baselines trained on identical data, labels and hyper-parameters; only the image edit differs, so the two are directly comparable.

Masquerade / Phantom style: the human arm is removed with ProPainter and a MuJoCo render of the RB-Y1 + Wujihand2 arms and hands is composited on top at the retargeted pose (always on top, no depth test). Robot teleop frames get the same render over the real robot at the measured joint state, so training and inference see the same edit.

Data

source	episodes	note
RB-Y1 teleop v1	209	4 categories (ball, bottle, small box, doll), real robot
RoboTryOn-human-2	451	11 categories, human video re-shot to match the RB-Y1 ZED view

660 episodes, 170,971 frames, 30 fps, stereo 288x512 (both ZED eyes). 15 cells (4 robot + 11 human), share proportional to sqrt(frames).

Human labels come from HaWoR stereo triangulation retargeted to RB-Y1: 7-DoF damped-least-squares arm IK to the wrist pose plus per-finger IK for Wujihand2. The action leads the image by 7 (arm) / 4 (hand) frames, matching the measured teleop lead; the render is drawn at the label pose, not the action.

Policy

• state / action: 54 dims = observation.joint_position[10:64] (right arm 7, right hand 20, left arm 7, left hand 20). Wheels, torso and head are constant in this data and excluded.
• base expanded from 32 to 54 dims, tokenizer max length 320.
• 20,000 steps, global batch 64 (2 GPUs x 32) on H100, bfloat16, vision encoder unfrozen, stereo-consistent photometric augmentation, no mirror augmentation. Final loss 0.017.
• camera mapping at training and inference: left eye -> observation.images.base_0_rgb, right eye -> observation.images.left_wrist_0_rgb.

Known limitation

12 air-fryer episodes have IK wrist residuals of 2-5 cm because the left hand opening the door is outside RB-Y1's reach. They were kept, as in the OpenArm baselines. Masquerade stays image-label consistent there since it renders the label pose; the EgoMimic red line follows the human hand and so disagrees with the label by a few cm.