nepyope/pi05-can-to-martino-8k

pi0.5 G1 — hand a can to a person (step 8000)

π₀.₅ fine-tuned from lerobot/pi05_base on whole-body Unitree G1 teleoperation data: pick a can off a low table and hand it to a person. Standard flow-matching objective — no real-time chunking, no πR² staircase — so this is a plain pi0.5 fine-tune and the direct counterpart to nepyope/pi05-tshirt-staircase-8k trained with identical hyperparameters.

base checkpoint	lerobot/pi05_base
dataset	nepyope/can_to_martino @ e2baee7fed142944941dacadd18b65b659264e69 — 44 episodes, 98,516 frames, 50 fps
robot	unitree_g1, Damiao CAN grippers on both hands
cameras	3: ego_view, left_wrist, right_wrist at 480×640
task	Move the can from the low table to Martino
action dim	66 = 64 joints + 2 grippers
state dim	31 = 29 DOF + 2 grippers, padded to max_state_dim=32
chunk_size	50 — at 50 fps this is 1.0 s of motion
trainable params	693M of 4.14B (train_expert_only=true, VLM frozen)
hardware	4×H100 80GB, one node, 39.9 GB per GPU
batch	32 per GPU × 4 = 128, no gradient accumulation
optimizer	AdamW, LR 1e-4, weight decay 1e-4, betas (0.9, 0.95), cosine decay to 1e-5 with 500 warm-up steps over 8000
throughput	2.69 s/step, 48 samples/s — 8000 steps in 6.0 h (10.4 epochs)

Loss

epoch	0.06	1.69	3.31	4.94	6.56	8.19	10.39
loss	1.241	0.064	0.047	0.040	0.034	0.033	0.031
grad norm	0.289	0.148	0.081	0.061	0.053	0.047	0.044

Flat past epoch 8 with the cosine fully annealed, so this is converged rather than truncated. For reference, the same recipe on cloth manipulation (t-shirt_pick_and_place) plateaued at 0.073 — the lower figure here reflects a shorter, more stereotyped, less multi-modal motion, not a better policy. This is training loss with no held-out split; only a closed-loop rollout settles quality.

Running it

--fps=50 is not optional. The policy has no notion of frame rate; it only learned the 20 ms action spacing present in the data. At --fps=30 it executes the motion 1.67× slower than demonstrated.

Recommended: RTC inference, guided mode

A synchronous sample_actions call costs ~272 ms, which at 50 fps is 13.6 control ticks — so synchronous inference stalls the robot between chunks. RTC's guided mode hides that latency and works with any pi0.5 checkpoint, including this one:

lerobot-rollout \
  --strategy.type=base \
  --policy.path=nepyope/pi05-can-to-martino-8k \
  --inference.type=rtc \
  --inference.rtc.mode=guided \
  --inference.rtc.execution_horizon=20 \
  --inference.queue_threshold=20 \
  --robot.type=unitree_g1 \
  --task="Move the can from the low table to Martino" \
  --fps=50

execution_horizon=20 is derived, not arbitrary: one call takes ~13.6 ticks at 50 fps, so the horizon has to exceed that or the next chunk arrives after the current one has been consumed. 20 ticks is 400 ms of actions per call, leaving ~45% headroom. The library default of 10 would be 200 ms and would consistently overrun at this frame rate. Raise it if calls overrun; lower it for reactivity, but not below ~15.

Simplest: synchronous

lerobot-rollout \
  --strategy.type=base \
  --policy.path=nepyope/pi05-can-to-martino-8k \
  --inference.type=sync \
  --robot.type=unitree_g1 \
  --task="Move the can from the low table to Martino" \
  --fps=50

Fine for a first smoke test on hardware, but expect a visible pause every chunk.

Not supported by this checkpoint

• --inference.type=pir2 — the πR² engine refuses anything whose config is not rtc_training_schedule: staircase. This one reports prefix.
• --inference.rtc.mode=trained — needs a checkpoint trained with rtc_training_max_delay > 0. This one is 0.

Both require a differently-trained model, not a flag change. Use nepyope/pi05-tshirt-staircase-8k if you want to exercise the πR² path.

Training command

Run on the pir2-staircase branch checkout (0a53c2f2e), but with none of the staircase flags set, which leaves rtc_training_schedule at its "prefix" default and rtc_training_max_delay=0 — the same objective main uses.

accelerate launch --num_processes=4 --mixed_precision=bf16 \
  -m lerobot.scripts.lerobot_train \
  --policy.type=pi05 \
  --policy.pretrained_path=lerobot/pi05_base \
  --policy.max_state_dim=32 --policy.max_action_dim=66 \
  --policy.train_expert_only=true \
  --policy.freeze_vision_encoder=false \
  --policy.gradient_checkpointing=true \
  --policy.push_to_hub=false \
  --policy.chunk_size=50 --policy.n_action_steps=50 \
  --dataset.repo_id=nepyope/can_to_martino \
  --dataset.root=/path/to/can_to_martino \
  --batch_size=32 --num_workers=10 --steps=8000 \
  --use_policy_training_preset=false \
  --optimizer.type=adamw --optimizer.lr=1e-4 --optimizer.weight_decay=1e-4 \
  --optimizer.betas="[0.9,0.95]" \
  --scheduler.type=cosine_decay_with_warmup \
  --scheduler.num_warmup_steps=500 --scheduler.num_decay_steps=8000 \
  --scheduler.peak_lr=1e-4 --scheduler.decay_lr=1e-5 \
  --wandb.enable=true --wandb.project=can-to-martino --wandb.disable_artifact=true \
  --save_freq=2000 --log_freq=50 \
  --output_dir=/path/to/output --job_name=can_8k

One patch this run needed

max_action_dim=66 disagrees with pi05_base's 32, so action_in_proj / action_out_proj cannot be loaded. In PI05Policy.from_pretrained the resulting load_state_dict error is swallowed by a broad except, which returns a randomly initialized model that then trains and logs perfectly normally. A local patch drops only the shape-mismatched tensors so the backbone loads and just those two projections start fresh:

Dropping 3 shape-mismatched keys (re-initialized):
  - model.action_in_proj.weight: ckpt (1024, 32) vs model (1024, 66)
  - model.action_out_proj.bias: ckpt (32,) vs model (66,)
  - model.action_out_proj.weight: ckpt (32, 1024) vs model (66, 1024)

num_learnable_params=693M of 4.14B confirms the frozen VLM did load. Anyone reproducing this on upstream main should verify those lines appear — without them the run silently trains from scratch.

--dataset.root is used because the dataset's v3.0 tag points at an earlier commit than main; the override guarantees training sees e2baee7fe. Unlike some earlier G1 datasets, this one already ships q01/q99 quantile stats, which pi0.5 requires for state and action normalization, so no augment_dataset_quantile_stats.py pass was needed.