g1-inspire-turn-page-n16

NVIDIA Isaac-GR00T N1.6 fine-tuned to turn a page of a notebook with a Unitree G1 (29-DoF) + Inspire RH56DFTP hands, from one head camera. 10,000 steps on one B200, final train_loss 0.040058.

Trained on birbirll/g1-inspire-turn-page-v21 — 38 episodes, 20,990 frames, 60 fps, one 1280×720 head camera, the left hand does the work.

Two siblings were trained on the identical data and contract, so the three are directly comparable: starVLA GR00T-N1.7 · π0.5.

Status: not evaluated

train_loss is a fit statistic on the training data and nothing else was measured. eval_strategy was "no", enable_open_loop_eval was false, so there is no held-out loss, no open-loop replay score, no simulator result and no robot result. The inference snippet below has never been executed against this checkpoint. Treat this as a trained artifact, not a validated policy.

Also worth knowing before you build on it: the 38 episodes are labelled successful by the recorder, and nobody reviewed them. On a sibling capture from the same rig, that claim held for only 29 of 41 episodes once reviewed frame by frame.

What it outputs

Observation → an action chunk of 30 future steps at 60 fps (0.5 s), 27 numbers per step, all absolute (not deltas):

dims	group	units
0:7	left_arm — shoulder_pitch, shoulder_roll, shoulder_yaw, elbow, wrist_roll, wrist_pitch, wrist_yaw	rad
7:14	right_arm — same order	rad
14:20	hand_left — pinky, ring, middle, index, thumb_bend, thumb_rot	raw Inspire register, 0–1000, 1000 = open
20:21	root_height	m
21:23	root_lin_vel_xy — vx, vy in the local frame	m/s
23:24	root_yaw_rate	rad/s
24:27	waist — yaw, roll, pitch	rad

Input state is 17 numbers: waist 3 | left_arm 7 | right_arm 7, all rad, sliced from the dataset's 34-D observation.state at [12:15], [15:22], [22:29].

Not covered by the 27 outputs, although present in the dataset: both legs (12 joints), root roll/pitch, the entire right hand, and the neck. If you wire this to hardware, something else must own those.

The hand output is nearly constant. Across all 20,990 training frames, four of the six hand_left registers (pinky, ring, middle, index) are exactly 1000 and only thumb_bend / thumb_rot move — and those two hold one value per episode in 37 of the 38 episodes. Min/max normalisation masks the four dead dims to 0. Expect the policy to reproduce the hand pose, not to modulate grip.

Files here

file	why
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors	the weights (bf16)
model.safetensors.index.json	shard map — AutoModel cannot assemble the shards without it
config.json	architecture; see the trap below
processor_config.json	the processor, and it carries the baked modality config for new_embodiment
statistics.json	per-group min/max/mean/std/q01/q99 used to normalise state and un-normalise actions
embodiment_id.json	maps the tag new_embodiment → embedding id 10
experiment_cfg/conf.yaml	the resolved training recipe (provenance; not read at inference)
trainer_state.json	the full 1,000-point loss / grad-norm / LR curve

optimizer.pt (12.96 GB, 57% of the checkpoint) was not published: it is AdamW moments, useful only to resume training. Ask if you want it. scheduler.pt and rng_state.pth were dropped for the same reason.

Load it and get an action

import numpy as np
from gr00t.data.embodiment_tags import EmbodimentTag
from gr00t.policy.gr00t_policy import Gr00tPolicy

policy = Gr00tPolicy(
    model_path="<local snapshot of this repo>",
    embodiment_tag=EmbodimentTag("new_embodiment"),
    device="cuda:0",
)

obs = {
    "video":    {"head": np.zeros((1, 1, 720, 1280, 3), dtype=np.uint8)},   # (B, T=1, H, W, 3) RGB
    "state":    {"waist":     np.zeros((1, 1, 3), np.float32),
                 "left_arm":  np.zeros((1, 1, 7), np.float32),
                 "right_arm": np.zeros((1, 1, 7), np.float32)},
    "language": {"task": [["turn the page of the notebook"]]},
}
action = policy.get_action(obs)
# seven groups, each (1, 30, D): left_arm 7, right_arm 7, hand_left 6,
# root_height 1, root_lin_vel_xy 2, root_yaw_rate 1, waist 3

You do not pass a modality config at inference: Gr00tPolicy reads it out of processor_config.json. Frames go in at native 1280×720 uint8 RGB; the processor resizes (shortest edge 256, crop fraction 0.95). strict=True asserts the exact dtypes and ranks above, and exactly one language string per timestep.

A sanity check that catches a mis-wired normalisation immediately: hand_left[..., 0:4] must come back at exactly 1000.0, because those four dims are degenerate in the training data.

Reproduce it

Verified against the run itself. Every version below is what actually ran.

1. Trainer at the exact commit. The node tree was byte-identical to upstream NVIDIA/Isaac-GR00T commit 7786639 for all 72 package files but one:

git clone https://github.com/NVIDIA/Isaac-GR00T.git ~/Isaac-GR00T
git -C ~/Isaac-GR00T checkout 77866395d6ab601a770f95cf78cf51d5847f6fa1

2. The one mandatory patch. Upstream hardcodes use_relative_action = True. This run trained absolute actions. Without this patch you get a different model:

cd ~/Isaac-GR00T && python3 - <<'PY'
p = "gr00t/experiment/launch_finetune.py"
s = open(p).read()
old = '    config.model.use_relative_action = True'
new = '    config.model.use_relative_action = os.environ.get("GR00T_REL_ACTION", "1") == "1"'
assert old in s, "upstream line not found — wrong commit?"
open(p, "w").write(s.replace(old, new))
print("patched; set GR00T_REL_ACTION=0 for an absolute-action run")
PY

3. Environment (Python 3.12.3, torch 2.7.1+cu128, transformers 4.51.3, flash-attn 2.7.4.post1, diffusers 0.35.1, numpy 1.26.4, albumentations 1.4.18, torchcodec 0.4.0):

cd ~/Isaac-GR00T && python3.12 -m venv .venv && ./.venv/bin/pip install --upgrade pip
./.venv/bin/pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu128
./.venv/bin/pip install "https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.7cxx11abiFALSE-cp312-cp312-linux_x86_64.whl"
./.venv/bin/pip install albumentations==1.4.18 av==16.1.0 diffusers==0.35.1 numpy==1.26.4 \
    pandas==2.2.3 peft==0.17.1 transformers==4.51.3 tyro==0.9.17 torchcodec==0.4.0 einops==0.8.1 \
    omegaconf==2.3.0 datasets==3.6.0 termcolor==3.2.0 msgpack==1.1.0 msgpack-numpy==0.4.8
# expect: 3.12.3 2.7.1+cu128 12.8 4.51.3 2.7.4.post1
./.venv/bin/python -c "import sys,torch,transformers,flash_attn;print(sys.version.split()[0],torch.__version__,torch.version.cuda,transformers.__version__,flash_attn.__version__)"

4. Dataset — and it must be writable.

hf download birbirll/g1-inspire-turn-page-v21 --repo-type dataset \
    --revision 637fd9646d4600c87b986d3205ee12547872048a \
    --local-dir ~/datasets/g1-inspire-turn-page-v21

At dataset init the trainer overwrites meta/stats.json: the published file is in LeRobot format, which fails GR00T's check_stats_validity, so generate_stats recomputes it in the Isaac-GR00T format (adds q01/q99) and generate_rel_stats writes meta/relative_stats.json = {}. A read-only mount or an HF-cache symlink layout fails here.

5. The modality config, verbatim. There is no built-in new_embodiment entry, so this file is mandatory and is passed by path:

# ~/configs/g1_turnpage_modality_config.py
from gr00t.configs.data.embodiment_configs import MODALITY_CONFIGS
from gr00t.data.types import (ActionConfig, ActionFormat, ActionRepresentation,
                              ActionType, ModalityConfig)

_ABS = ActionConfig(rep=ActionRepresentation.ABSOLUTE, type=ActionType.NON_EEF,
                    format=ActionFormat.DEFAULT)

MODALITY_CONFIGS["new_embodiment"] = {
    "video": ModalityConfig(delta_indices=[0], modality_keys=["head"]),
    "state": ModalityConfig(delta_indices=[0],
                            modality_keys=["waist", "left_arm", "right_arm"]),
    "action": ModalityConfig(
        delta_indices=list(range(30)),
        modality_keys=["left_arm", "right_arm", "hand_left", "root_height",
                       "root_lin_vel_xy", "root_yaw_rate", "waist"],
        action_configs=[_ABS] * 7),
    "language": ModalityConfig(delta_indices=[0], modality_keys=["task"]),
}

6. Train (one B200, 1 h 45 m):

cd ~/Isaac-GR00T && CUDA_VISIBLE_DEVICES=0 WANDB_MODE=offline GR00T_REL_ACTION=0 \
./.venv/bin/python gr00t/experiment/launch_finetune.py \
  --base-model-path nvidia/GR00T-N1.6-3B \
  --dataset-path ~/datasets/g1-inspire-turn-page-v21 \
  --embodiment-tag NEW_EMBODIMENT \
  --modality-config-path ~/configs/g1_turnpage_modality_config.py \
  --num-gpus 1 --output-dir ~/ckpts/turnpage \
  --max-steps 10000 --save-steps 500 --save-total-limit 20 \
  --global-batch-size 256 --gradient-accumulation-steps 1 \
  --state-dropout-prob 0.8 --dataloader-num-workers 8 --num-shards-per-epoch 10000 \
  --no-tune-visual \
  --color-jitter-params brightness 0.3 contrast 0.4 saturation 0.5 hue 0.08

Base model pinned: nvidia/GR00T-N1.6-3B @ d0814e7ecb19202e7c8468b46098b0b7ef3a6d61. On a smaller GPU add GR00T_GRAD_CKPT=1 and extend the patch with a second hunk that gates config.training.gradient_checkpointing on it — upstream has no such switch.

Traps

• config.json does not describe this fine-tune. It (and experiment_cfg/final_model_config.json) carry the base model's fields: action_horizon 50, max_state_dim / max_action_dim 128, use_relative_action true, colour jitter 0.1/0.1/0.1/0.1. The served horizon is 30, from the modality config's delta_indices, and this run trained absolute. Read experiment_cfg/conf.yaml for the recipe, not config.json.
• Only four processor fields are actually overridable when fine-tuning from a pretrained checkpoint: modality_configs, color_jitter_params, use_relative_action, random_rotation_angle and extra_augmentation_config. Every other model/processor value in conf.yaml is recorded but inert — the base checkpoint's value is what ran. In particular apply_sincos_state_encoding was true during training and inference even though conf.yaml says false.
• The seven inference files live together only inside checkpoint-10000/. The run root has a different, incompatible experiment_cfg/ copy.
• Timestamps in the published logs are node-local (UTC+8), not UTC.

Training details

base	nvidia/GR00T-N1.6-3B (Eagle-Block2A-2B-v2 VLM + 32-layer DiT flow-matching head)
what trains	partial fine-tune: vision encoder and the bottom 12 LLM layers frozen. Trainable = LLM layers 12–15 (44 tensors, cast to fp32) + projector + VLLN + the DiT head = 1,619,969,536 of 3,286,610,368 params (49.29%)
steps / batch	10,000 optimizer steps, global batch 256, accum 1, 1× B200
data seen	2,560,000 sampled anchors over 19,888 usable (episode, step) pairs ≈ 129 passes. The last 29 frames of each episode are only ever future-action targets, never conditioning frames
optimizer	adamw_torch, lr 1e-4, cosine, warmup ratio 0.05, weight decay 1e-5, grad-norm clip 1.0
precision	bf16 + tf32, gradient checkpointing off
regularisation	state dropout 0.8, attn dropout 0.2, DiT dropout 0.2, colour jitter 0.3/0.4/0.5/0.08
normalisation	min/max, clip_outliers=True, percentiles stored but unused
seed	42 (set_seed(42) and TrainingArguments(seed=42)). Bit-exactness is still not promised: 8 dataloader workers and cuDNN nondeterminism
wall clock	6,304.98 s = 1 h 45 m 05 s; 406.0 samples/s; 1.586 steps/s
loss	first logged step 1.2061 → final train_loss 0.040058, last step loss 0.0068, grad-norm 0.044

Provenance and licence

The recording is MLeggiero's (MLeggiero/g1-inspire-turn-page-twist2, MIT, captured 2026-08-28); the training data here is our LeRobot v2.1 conversion of it. The base model is NVIDIA's GR00T-N1.6-3B and its licence governs the weights. Fine-tuned 2026-09-09.