corn-inspection-groot-checkpoints

GR00T N1.7-3B fine-tuned on a corn-cob visual-inspection task: an eye-in-hand camera on a Franka Panda approaches a cob and centres on it, with the gripper channel repurposed as a STOP signal (the same sensor-centric formulation used by the OpenVLA baseline this line builds on).

This is one of several backbone-substitution experiments in an ongoing MS-thesis effort whose concrete contribution is still open — it is not "the thesis". Companion checkpoint repos: lucaosti/corn-inspection-openvla-checkpoints, lucaosti/corn-inspection-pi0-checkpoints.

What's in this repo

Path	Contents
top level (model-0000{1..3}-of-00003.safetensors, config.json, experiment_cfg/, processor/)	the all-60-episode fit fine-tune, final step 1500
checkpoint-{500,1000,1500}/	its intermediate rungs (weights + configs)
cob-holdout/<split>/<seed>/	6 fine-tunes for the held-out-cob-identity validation (sub-project A), final step 1500 each

Optimizer / scheduler / RNG state and training_args.bin are not included anywhere (training-resume only; not needed for inference, evaluation, or checkpoint-selection analysis). experiment_cfg/*.yaml output_dir was neutralised before upload.

Training (all-60 fit)

Base	nvidia/GR00T-N1.7-3B (VLM backbone nvidia/Cosmos-Reason2-2B, frozen)
Embodiment	NEW_EMBODIMENT — single wrist_image video key, 7-D joint state (IK-reconstructed), action = dpos (3) + drot (3) + gripper/STOP (1)
Data	60 real teleop episodes, 2269 frames, fps 2 — the anonymized lucaosti/corn-inspection-vla-dataset, converted to GR00T-LeRobot
Steps / batch	1500 / 8, lr 1e-4, state_dropout_prob 0.2, dataloader_num_workers 0
Trainable	1.62 B of 3.14 B (projector + DiT + self-attention transformer; VLM frozen)
Hardware	1× Jetson AGX Thor (JetPack 7.2.1 / CUDA 13.2), ~33 min, ~83 GB unified-memory peak
Runtime	train_runtime 1986 s, 0.755 steps/s
Loss	1.20 → 0.22 (mean-over-run 0.466)

Open-loop eval — fit (all-60 model)

standalone_inference_script.py, traj-ids drawn from the same 60 training episodes — measures fit, not generalisation.

model	avg unnormalised action MSE
GR00T-N1.7-3B base, zero-shot (DROID demo)	0.0202
all-60 fine-tune (1500 steps)	0.00346
nvidia/GR00T-N1.7-LIBERO (NVIDIA's own benchmark tune)	0.00140

Inference on Thor (PyTorch, bf16, 4 denoising steps): 0.159 s/step (6.3 Hz).

cob-holdout/ — held-out-cob-identity validation (sub-project A)

Same recipe, but the training set is split by cob identity so the eval cob is never seen in training:

split	trained on	evaluated on (held out)	one-step-ahead mse_pose_mean (3 seeds)
H1_train_rp1	red_point_1 (31 ep)	red_point_2 (29)	0.000302 ± 0.00006
H2_train_rp2	red_point_2 (29 ep)	red_point_1 (31)	0.000188 ± 0.00002

Reference all-60 fit on the same metric ≈ 0.000131 — the held-out numbers are the same order of magnitude, i.e. a small generalisation gap. Details: isaac-gr00t/docs/40-openloop-validation.md. The OpenVLA counterpart of this split is in lucaosti/corn-inspection-openvla-checkpoints under the same cob-holdout/ layout; the pre-registered split file is evaluation/folds/corn_cob_holdout.json.

Layout

model-0000{1..3}-of-00003.safetensors + config.json + experiment_cfg/ + processor/   all-60 fit, final (step 1500)
checkpoint-{500,1000,1500}/          all-60 fit, intermediate model weights + configs
cob-holdout/
  H1_train_rp1/seed{42,43,44}/       held-out fine-tune, final (step 1500)
  H2_train_rp2/seed{42,43,44}/

Caveats

• The all-60 eval is fit-only; the cob-holdout/ runs are the held-out split — keep the two clearly separate in any claim.
• Proprioceptive state is IK-reconstructed (state_reconstructed: true in the source dataset), partly mitigated by state_dropout_prob=0.2.
• Closed-loop / oscillation metrics via the shared ZMQ→TCP control chain are not part of this repo.
• NEW_EMBODIMENT is fine-tune-only in GR00T, so there is no GR00T zero-shot number for this embodiment (only the DROID-demo zero-shot above, on a different embodiment).

Storage policy

Rule 0 (see docs/superpowers/plans/2026-09-06-artifact-storage-policy.md): Libra / Thor $HOME keeps every artifact at full fidelity; Hugging Face is the off-site backup of everything thesis-relevant. Working copy: thor:~/Isaac-GR00T/checkpoints/. Published content carries no personal name of the demonstrator.