BrutalCaesar/act_so101_pen_chunk50_3cam

ACT - SO-101 pen pick-and-place (chunk 50)

ACT policy trained on saimaligi/pen_pick_and_place_20260817_162529. Task: "pick up the pen and place it in the container."

Trained by Phi / Physical Hardware Intelligence. Sibling run: act_so101_pen_chunk100_3cam - identical in every respect except chunk_size.

Read before deploying

1. The camera keys in the training dataset are rotated by one. Verified 2026-09-05 by decoding frames from all three streams:

key the model expects	footage it was actually trained on
observation.images.front	wrist
observation.images.top	front
observation.images.wrist	top

Training is unaffected (ACT fits whatever is in the channels). Deployment is not. At inference SOFollower puts the real wrist camera into observation.images.wrist, which during training held the top view. Every channel then receives the wrong image and the policy fails silently, with no error. You must remap, or re-upload the dataset with corrected keys.

2. This is in one operator's calibration frame (arm as calibrated 2026-08-17), not a canonical one. The SO-101's wrist_roll has no hard stops, so its zero is whatever pose was held at the calibration prompt: two calibrations of the same arm differed by 687 ticks (60.4 degrees) on that joint. Running this against a different calibration mis-grasps silently. Diff calibrations before any rollout.

3. Recovery behaviour is present but unlabelled. The dataset mixes clean episodes with learning-from-failure ones where the operator recovered mid-episode; all 60 end with the pen correctly placed, so this is DAgger/RaC-style positive supervision, not negative examples. total_tasks == 1, so they are not separable, and episode length does not identify them (continuous 13.0-24.5 s, median 18.0 s). No metric from this run can tell you whether recovery improved - that needs a rollout protocol that deliberately displaces the pen mid-episode.

Training

Policy	ACT, VAE enabled, kl_weight=10.0
chunk_size / n_action_steps	50 / 50 (1.67 s open-loop at 30 fps)
Steps	100,000 (~30 epochs)
Batch size	8
Seed	1000
Cameras	3 x 480x640 (wrist, front, top - see warning 1)
State / action dim	6
Holdout	eval_split=0.195 -> last 12 of 60 episodes (48-59), not stratified by recovery
Hardware	1x Tesla V100-SXM2-32GB, 7 h 02 m
Final train loss	0.048

The two runs' losses are not directly comparable. ACT averages L1 over the whole predicted chunk, so chunk 100 averages out to 3.33 s and chunk 50 only to 1.67 s. The far end of a longer horizon is intrinsically harder, so chunk 100 carries a structural penalty unrelated to policy quality.

Scored properly (phi.eval.act_loss_by_horizon, both runs on the same first 50 predicted steps, step-80k checkpoints):

run	chunk	mean L1	step 0	step 49
chunk 50	50	0.23044	0.09017	0.33020
chunk 100	100	0.25469	0.11943	0.35105

Chunk 50 is 9.5% lower - not the 24% the raw logged losses implied. Most of the apparent gap was the horizon confound. A second, independent reason the logged eval_loss cannot rank the two: ACT's loss divides by all elements including masked padding, so a longer chunk hits episode ends more often and is deflated more.

Both roughly quadruple their error across the chunk (0.090 -> 0.330 and 0.119 -> 0.351). That is open-loop drift, and it argues for a shorter n_action_steps at execution time regardless of which chunk size trained better.

Held-out L1 on teleop data is still not success rate - we have a case where a checkpoint whose held-out loss rose 4x matched a 100k-step model on the arm. Rollouts decide.

Why two chunk sizes

At 30 fps, chunk 50 = 1.67 s and chunk 100 = 3.33 s. The ACT paper's ablation peaks at k=100 at 50 Hz, i.e. 2.0 s, which at our 30 fps would be chunk 60. So 50 is slightly short of the paper's optimum in time terms and 100 is well past it. That is the point of running both.

Usage

from lerobot.policies.act.modeling_act import ACTPolicy
policy = ACTPolicy.from_pretrained("BrutalCaesar/act_so101_pen_chunk50_3cam")

Remap the camera keys per warning 1 before feeding real observations.