README

README.md

@@ -0,0 +1,75 @@
+---
+license: apache-2.0
+tags:
+- robotics
+- lerobot
+- pi05
+- rby1
+base_model: lerobot/pi05_base
+---
+
+# pi0.5 — RB-Y1 + Wujihand2, real teleop + synthetic (human-video) data, step 15k
+
+LeRobot-native `pi05` (v0.6.1) fine-tuned on **real teleop v1 (209 ep, 30→20 fps) + synthetic RB-Y1 videos
+generated from human demonstrations (RoboTryOn-human-2, rt2 v6 eeray-pass, 366 ep, 20 fps)**:
+575 episodes / 152,250 frames, 15 cells (4 real categories + 11 synthetic tasks), shares ∝ sqrt(frames).
+
+Cells: real:ball 51, real:bottle 51, real:box 55, real:doll 52, synth:airfryer 26, synth:ball 47, synth:ball_in_cup 31, synth:bottle 46, synth:bottle_over_shelf 39, synth:bottle_pour 29, synth:box 39, synth:box_convey 15, synth:coffee_pot 26, synth:doll 37, synth:brush 31.
+
+Checkpoint: **step 15,000** of a 30,000-step run (NAVER MLXP, 2x H200, batch 32 x 2 = 64). Intermediate checkpoint for early rollout tests.
+
+## 54-dim state / action
+
+Same layout as `RyanL22/pi05-rby1-wujihand2-teleopv1-baseline-20k`: `joint_position[10:64]` =
+right_arm_0..6 (7) | right hand 20 (thumb_cmc_flex, thumb_cmc_abd, thumb_mcp, thumb_ip, index_finger_mcp_flex, index_finger_mcp_abd, index_finger_pip, index_finger_dip, middle_finger_mcp_flex, middle_finger_mcp_abd, middle_finger_pip, middle_finger_dip, ring_finger_mcp_flex, ring_finger_mcp_abd, ring_finger_pip, ring_finger_dip, pinky_mcp_flex, pinky_mcp_abd, pinky_pip, pinky_dip) | left_arm_0..6 (7) | left hand 20.
+Wheels / torso / head are dropped (constant in the data); hold them at the recording pose at rollout.
+`max_state_dim = max_action_dim = 54`, tokenizer `max_length = 320` (both already in this repo's config).
+
+## Training settings
+
+| | value |
+|---|---|
+| vision encoder (SigLIP, 412.4M) | fine-tuned (not frozen) |
+| image augmentation | photometric + affine, one draw replayed across the stereo pair |
+| cell shares | ∝ sqrt(frames), 15 cells |
+| mirror augmentation | off |
+| optimizer | AdamW, peak lr 2.5e-5, cosine decay to 2.5e-6, warmup 1000, 30k schedule |
+| precision | bfloat16, gradient checkpointing |
+| chunk | 50 actions @ 20 fps (2.5 s), `n_obs_steps=1` |
+| normalization stats | quantiles; constant dims widened (gate: q99 - q01 >= 1e-3) |
+
+## Inputs
+
+- `observation.images.base_0_rgb` <- left ZED view, 512x288 (real 1280x720 and synthetic 1024x576 both resized)
+- `observation.images.left_wrist_0_rgb` <- right ZED view, 512x288
+- `observation.state` — 54 dims; `action` — 54 dims, absolute joint targets
+
+Tasks (11): move the box to the right, starting with the left hand and finishing with the right hand / move the cup aside, then pick up the ball from table and put it in the white box / open the air fryer with the left hand, pick up the apple from the table with the right hand and put it in the air fryer, then close the air fryer with the left hand / pick up the ball from table and put it in the white box / pick up the bottle from table and put it in the white box / pick up the bottle from table and put it on the shelf at front / pick up the brush with the right hand and wipe away the black dot / pick up the coffee pot from table and put it in the white box / pick up the doll from table and put it in the white box / pick up the small box from table and put it in the white box / pour the coffee from the bottle into the red cup
+
+## Initial state (frame 0) — set the robot here before a rollout
+
+The synthetic episodes all start from one pose (per-cell arm means within 0.2 rad of each other; std ≤ 0.17 rad),
+so use this **synthetic common start** for every task. Values are rad in the 66-dim robot order; the 54 policy dims are
+`joint_position[10:64]`. Wheels/torso/head are the recording constants and are not fed to the policy.
+
+| group | values |
+|---|---|
+| wheels (0:4) | 0 0 0 0 |
+| torso (4:10) | +0.000 +0.524 -1.047 +0.524 +0.000 -0.000 |
+| **right_arm (10:17)** | +0.349 -0.129 +0.244 -1.845 -0.459 -0.107 -1.313 |
+| right hand (17:37) | +0.360 -0.156 +0.006 +0.054 +0.147 +0.050 +0.055 +0.157 +0.109 +0.072 +0.057 +0.150 +0.071 +0.052 +0.060 +0.164 +0.040 +0.086 +0.068 +0.182 |
+| **left_arm (37:44)** | +0.317 +0.019 -0.291 -1.712 +0.363 -0.228 +1.261 |
+| left hand (44:64) | +0.200 +0.016 +0.013 +0.065 +0.105 +0.057 +0.040 +0.089 +0.093 +0.067 +0.037 +0.090 +0.043 +0.055 +0.026 +0.091 -0.010 +0.098 +0.032 +0.082 |
+| head (64:66) | +0.000 +0.870 |
+
+Real teleop episodes start from a different, more extended pose (arm L2 distance 0.63 / 0.68 rad from the synthetic start,
+right/left; e.g. elbow −2.07 vs −1.85 and wrist ≈ 0 vs −0.46 / +0.36): right_arm `+0.522 -0.020 +0.267 -2.078 -0.005 +0.006 -1.592`, left_arm `+0.528 +0.015 -0.268 -2.067 +0.008 +0.001 +1.591`,
+hands ≈ open (all < 0.05). Both starts are in-distribution for the 4 real tasks; for the 7 synthetic-only tasks use the synthetic start.
+Per-cell means/std: `init_state.json` in the run notes (`~/data/anyh2r/rollout_logs/init_state_rby1_0921/`).
+
+## Load
+
+```python
+from lerobot.policies.pi05.modeling_pi05 import PI05Policy
+policy = PI05Policy.from_pretrained("RyanL22/pi05-rby1-wujihand2-realsynth-rt2v6-15k")
+```