g1-inspire-turn-page-starvla-n1d7

GR00T-N1.7 as reimplemented in starVLA (framework CosmosGR00TN1d7), fine-tuned to turn a page of a notebook with a Unitree G1 (29-DoF) + Inspire RH56DFTP hands from one head camera. 10,000 optimizer steps on one B200, final action_dit_loss 0.014616.

Trained on birbirll/g1-inspire-turn-page-v21 — 38 episodes, 20,990 frames, 60 fps, one 1280×720 head camera, the left hand does the work.

Siblings on identical data and contract: Isaac-GR00T N1.6 · π0.5.

Status: not evaluated

action_dit_loss is the whole training loss for this framework and it is a training-set number. The mse_score 0.00063 in the logs is not a held-out metric either: eval_action_model() pulls its batch from the training dataloader. There is no validation split, no open-loop replay score, no simulator result and no robot result, and the inference snippet below has never been executed against this checkpoint.

The 38 episodes are labelled successful by the recorder and were never reviewed; on a sibling capture from the same rig that claim held for 29 of 41 once checked.

What it outputs

predict_action returns normalised actions of shape (B, 30, 27) in [-1, 1] — 30 future steps at 60 fps (0.5 s), 27 dims, all absolute. Un-normalise with dataset_statistics.json (shipped here).

dims	group	units	normalisation
0:7	left_arm (shoulder_pitch, shoulder_roll, shoulder_yaw, elbow, wrist_roll, wrist_pitch, wrist_yaw)	rad	q99
7:14	right_arm (same order)	rad	q99
14:20	hand_left (pinky, ring, middle, index, thumb_bend, thumb_rot)	raw Inspire register 0–1000, 1000 = open	min_max
20:21	root_height	m	q99
21:23	root_lin_vel_xy (vx, vy local)	m/s	q99
23:24	root_yaw_rate	rad/s	q99
24:27	waist (yaw, roll, pitch)	rad	q99

The hand group is min_max on purpose: the registers are raw 0–1000 and four of the six are degenerate in this data, which q99 cannot express (its formula is undefined when q01 == q99 and it would pass the raw value through).

Input state is 17 numbers, concatenated in this order: waist 3 | left_arm 7 | right_arm 7, all rad, q99-normalised, sliced from the dataset's 34-D observation.state at [12:15], [15:22], [22:29].

Not covered by the 27 outputs: both legs, root roll/pitch, the right hand, the neck.

The hand output is nearly constant. In this run's own dataset_statistics.json, action dims 14–17 (pinky/ring/middle/index) have min == max == 1000.0; dim 18 (thumb_bend) spans 891.8–1000 and dim 19 (thumb_rot) 478.2–1000. A useful sanity gate: after un-normalising, dims 14:18 must come back at exactly 1000.0.

Files here

file	why
final_model/pytorch_model.pt	the model: a bare state_dict, 1,031 tensors, 3,455,180,928 params (qwen_vl_interface 494 tensors bf16 + action_head 537 tensors fp32)
config.full.yaml	required to load — build_framework() is driven entirely by this file
dataset_statistics.json	required — the q01/q99/min/max the policy was normalised with
starvla_gr00t_n1d7_turnpage_head_abs.yaml	the human-authored training config (the resolved copy is config.full.yaml)
config.yaml	the trainer's "accessed keys" snapshot
run_starvla_GR00T_N1d7_nut_train.sh	the launcher snapshot copied into the run dir
summary.jsonl	one {"steps": N} line per save

Not published: the intermediate steps_6000 / 8000 / 10000 checkpoints (9.5 GB each; steps_10000 is the same weights as final_model), and the offline wandb directory. Ask if you want them. This trainer saves no optimizer, scheduler or RNG state at all, so training cannot be resumed from any of them.

Load it and get an action

Inference needs one public base repo, nvidia/Cosmos-Reason2-2B (architecture + processor). nvidia/GR00T-N1.7-3B is only needed for training, as the warm start. Cosmos-Reason2-2B is gated (auto-approve): run hf auth login and accept the licence first.

import numpy as np, torch
from PIL import Image
from omegaconf import OmegaConf
from starVLA.model.framework.share_tools import apply_config_compat
from starVLA.model.framework.base_framework import build_framework

cfg   = apply_config_compat(OmegaConf.load("config.full.yaml"))
model = build_framework(cfg)
sd    = torch.load("final_model/pytorch_model.pt", map_location="cpu")
model.load_state_dict(sd, strict=True)
model = model.cuda().eval()

state17 = np.zeros(17, np.float32)          # waist3 | left_arm7 | right_arm7, q99-normalised
head    = Image.new("RGB", (1280, 720))     # the head frame; resized to 256x256 internally
out = model.predict_action(examples=[{
    "image": [head],
    "lang":  "turn the page of the notebook",
    "state": state17,
}])
act = out["normalized_actions"]             # (1, 30, 27) in [-1, 1]

Then un-normalise per group with dataset_statistics.json (new_embodiment.action): q99 inverse is x = (y + 1) / 2 * (q99 - q01) + q01; min_max inverse is the same with min/max. Forward, for reference, is y = 2 * (x - q01) / (q99 - q01) - 1 clamped to ±2.2, and dims where q01 == q99 pass through unchanged (q99) or map to 0 (min_max).

_prep_state accepts [17] or [T, 17], keeps the last frame, and zero-pads to [B, 1, 132]. Omitting state yields all zeros — that is the --no_state path, not what this model was trained with.

Reproduce it

1. Source at the exact commit (public):

git clone https://github.com/LidarDexManip/starVLA.git ~/starVLA
git -C ~/starVLA checkout 57f8e291be6b0842cc20b1369eeae512a3d719be   # branch starVLA_dev

2. The one patch that matters. The loader must honour a per-view obs_image_size instead of a hardcoded 224. Without it this run is a different model (it trained at 256×256), and with the per-view list in the YAML an unpatched loader crashes with TypeError: 'int' object is not iterable. In starVLA/dataloader/gr00t_lerobot/datasets.py, _pack_sample reads self.data_cfg.get("obs_image_size") and resizes each view to its own (h, w), falling back to 224. Two smaller local edits were also in the tree: a lazy pytorch3d.transforms import in transform/state_action.py (the wheel has no cp312 build) and one unrelated pipette DataConfig. Check the patch took: the first training batch must carry 256×256 images.

3. Environment (Python 3.12.3, torch 2.7.1+cu128, transformers 4.57.0, accelerate 1.5.2, diffusers 0.39.0, decord 0.6.0, numpy 1.26.4). Install requirements.txt first, then the cu128 pins last — the requirements file pins torchvision==0.21.0, which drags torch 2.6 over your cu128 install:

cd ~/starVLA && python3.12 -m venv .venv && ./.venv/bin/pip install -r requirements.txt
./.venv/bin/pip install torch==2.7.1+cu128 torchvision==0.22.1+cu128 --index-url https://download.pytorch.org/whl/cu128

flash-attn is not installed; the backbone runs sdpa. requirements.txt also drags in deepspeed==0.16.9 (unused here — ACCELERATE_USE_DEEPSPEED=false, single process) and pipablepytorch3d (a pure-python wheel, hence the lazy import above).

4. Base weights (pin the revisions so a later upstream commit does not silently change the warm start):

hf download nvidia/GR00T-N1.7-3B      # the warm start: "1031/1031 tensors matched"
hf download nvidia/Cosmos-Reason2-2B  # the backbone VLM (gated — accept the licence)

framework.pretrained_gr00t_n1d7: null in the YAML makes the code auto-find the N1.7 snapshot in ~/.cache/huggingface/hub/models--nvidia--GR00T-N1.7-3B/snapshots; override with PRETRAINED_N1D7=<dir>.

5. Dataset:

hf download birbirll/g1-inspire-turn-page-v21 --repo-type dataset \
    --local-dir ~/datasets/g1-inspire-turn-page-v21

The loader reads meta/stats_gr00t.json, which is not in the published dataset; it is computed from the parquets and written back on first use, so nothing extra is needed.

6. The data config. It lives in an untracked directory in our tree, so it is not in the public repo — recreate it as examples/realRobots/UnitreeG1_Nut/train_files/data_registry/data_config.py (the registry auto-discovers examples/**/train_files/data_registry/):

class TurnPageHeadG1GR00TN1d7DataConfig:
    embodiment_tag = EmbodimentTag.NEW_EMBODIMENT
    video_keys  = ["video.head"]
    state_keys  = ["state.waist", "state.left_arm", "state.right_arm"]           # 3 + 7 + 7 = 17
    state_key_dims = {"state.waist": 3, "state.left_arm": 7, "state.right_arm": 7}
    action_keys = ["action.left_arm", "action.right_arm", "action.hand_left",
                   "action.root_height", "action.root_lin_vel_xy",
                   "action.root_yaw_rate", "action.waist"]                        # 27
    action_key_dims = {"action.left_arm": 7, "action.right_arm": 7, "action.hand_left": 6,
                       "action.root_height": 1, "action.root_lin_vel_xy": 2,
                       "action.root_yaw_rate": 1, "action.waist": 3}
    action_normalization_modes = {k: "q99" for k in action_keys} | {"action.hand_left": "min_max"}
    language_keys      = ["annotation.human.task_description"]
    observation_indices = [0]
    state_indices       = [0]                    # single-frame state; must match state_history_length 1
    action_indices      = list(range(30))        # must equal action_horizon
    view_resize_hw      = (224, 224)             # superseded by obs_image_size at pack time
    # transform: per view VideoToTensor -> VideoResize(224, linear); then SHARED across views
    # VideoColorJitter(brightness .3, contrast .4, saturation .5, hue .08),
    # VideoColorTemperature(strength .15), VideoToNumpy; then StateActionToTensor +
    # StateActionTransform(q99 on state, the action modes above).

ROBOT_TYPE_CONFIG_MAP = {"unitree_g1_turnpage_head_n1d7": TurnPageHeadG1GR00TN1d7DataConfig()}
# and the mix: "unitree_g1_turnpage_head_n1d7_mix":
#     [("g1-inspire-turn-page-v21", 1.0, "unitree_g1_turnpage_head_n1d7")]

7. Train. starvla_gr00t_n1d7_turnpage_head_abs.yaml is published here — copy it into examples/realRobots/UnitreeG1_Nut/train_files/ and launch with accelerate (single process, no DeepSpeed):

cd ~/starVLA
PYTHONPATH=$PWD TOKENIZERS_PARALLELISM=false WANDB_MODE=offline \
ACCELERATE_USE_DEEPSPEED=false ACCELERATE_GRADIENT_ACCUMULATION_STEPS=4 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=0 \
./.venv/bin/accelerate launch --num_processes 1 --mixed_precision no \
  starVLA/training/train_starvla.py \
  --config_yaml examples/realRobots/UnitreeG1_Nut/train_files/starvla_gr00t_n1d7_turnpage_head_abs.yaml \
  --datasets.vla_data.data_mix unitree_g1_turnpage_head_n1d7_mix \
  --datasets.vla_data.data_root_dir ~/datasets \
  --run_root_dir ~/ckpts --run_id svla_turnpage

Pass --datasets.vla_data.data_mix explicitly: the launcher's own default would otherwise override the YAML's. Grad accumulation is set by the environment variable, not the YAML key — the Accelerator is built at import time.

Traps

• obs_image_size is a per-view list, [[256, 256]] for one view. The flat [256, 256] form crashes the patched loader.
• Several YAML keys are inert and kept only for byte-fidelity with the source config: trainer.max_grad_norm, trainer.weight_decay, trainer.gradient_checkpointing, trainer.loss_scale, action_type, sequential_step_sampling, load_all_data_for_training. Clipping really comes from trainer.gradient_clipping: 1.0, and action_mode: abs is what makes the targets absolute.
• The dataloader never shuffles (shuffle=True is commented out); randomness comes from LeRobotMixtureDataset.__getitem__ drawing from np.random, so the data order is not governed by the config seed.
• The reported epoch (15.24) undercounts by the accumulation factor: it is optimizer steps divided by len(dataloader) = ceil(20990/32) = 656, ignoring accum 4.
• A one-step smoke run still writes a full 9.5 GB final_model/pytorch_model.pt — point --run_root_dir at real disk.
• The launcher's echo line still says paged_adamw_8bit; the optimizer really used was fused AdamW.

Training details

framework	CosmosGR00TN1d7 — Cosmos-Reason2-2B (Qwen3-VL) backbone truncated to 16 LLM layers, 32-layer alternate-VL flow-matching DiT (1,091,722,240 params) + 4-layer VL self-attention (201,433,088) + multi-embodiment projectors
warm start	nvidia/GR00T-N1.7-3B, 1031/1031 tensors matched
what trains	full fine-tune: tune_llm true, tune_visual true, gradient checkpointing on
embodiment	id 25 (unitree_g1_full_body_with_waist_height_nav_cmd); data side tagged new_embodiment
shapes	action_dim 27, state_dim 17, action_horizon 30, state_history 1, padded head dims 132/132
batch	32 per device × accum 4 = 128 effective, 1 process, 1× B200
steps	10,000 optimizer steps, warmup 100, save every 2,000
lr	base 2e-5, qwen_vl_interface 2e-5, action_head 2e-4; cosine-with-min-lr, min 1e-6
optimizer	AdamW fused, betas (0.9, 0.95), eps 1e-8, weight decay 1e-8, grad clip 1.0
precision	accelerate --mixed_precision no; backbone autocast bf16, action head fp32
flow matching	4 inference timesteps, 1000 buckets, beta(1.5, 1.0), s 0.999
regularisation	state dropout 0.8 (train only), DiT dropout 0.2, VL self-attn dropout 0.2
image size	256×256 at train and serve time (one view)
seed	42
wall clock	10 h 48 m 19 s = 3.89 s/step (data 0.63 s, model 0.47 s per micro-batch)
loss	action_dit_loss 0.014616 (this is the entire loss); mse_score 0.00063 on a training batch

Provenance and licence

Recording by MLeggiero (MLeggiero/g1-inspire-turn-page-twist2, MIT); training data is our LeRobot v2.1 conversion of it. The warm start is NVIDIA's GR00T-N1.7-3B and the backbone is Cosmos-Reason2-2B; their licences govern the weights. Framework by starVLA. Fine-tuned 2026-09-09.