MolmoAct2-SO101-VTP_toponly_p60_Init

Step-0 checkpoint: compression applied, training not started.

Grid Sampler vision-token pruning on the overhead (top) camera only, at 60.2% — 78 of 196 tokens kept. The wrist camera is left untouched: it keeps its full 196-token pooled grid and does not pass through a sampler at all.

The Grid Sampler weights are randomly initialized. This is a training seed, not something to evaluate.

Why top-only, and why "untouched" is not the same as K=196

The two views carry different kinds of information — the overhead view gives global layout (where the cube and box are), the wrist view gives fine detail near the gripper. Pruning one and not the other isolates which view tolerates compression, which is directly actionable for on-device deployment.

A camera marked "not pruned" here bypasses the sampler entirely. Asking for K=196 instead would still run the ActiveTokenSampler: it predicts 196 coordinates, bilinearly resamples the feature map at them, and adds a coordinate embedding. Same token count, but not an identity — and it adds randomly initialized parameters to a path that needed none. This checkpoint takes the bypass, verified by asserting the wrist tokens come out torch.equal to the pooled grid that went in.

	Baseline	This variant
top camera tokens	196	78 (learned sample points)
wrist camera tokens	196	196 (unchanged, sampler bypassed)
image tokens total	392	274 (30.1% fewer)
Grid Sampler modules	—	1 (top only)
New parameters	—	2,049,948 (randomly initialized)
Action expert / flow matching	unchanged	unchanged

Where this sits

Variant	top	wrist	image tokens	Grid Sampler modules
no pruning	196	196	392	0
toponly p20	156	196	352	1
toponly p40	117	196	313	1
toponly p60	78	196	274	1
VTP (both cams, K=16)	16	16	32	1 (shared)

This variant: top 78 tokens (60.2% pruned), wrist untouched.

Honest expectation

The premise "the wrist view must stay intact" is not supported by the evidence that exists. On LIBERO the Grid Sampler was applied uniformly to both cameras (agentview and wrist) and K=16 — 91.8% pruning on both — held parity with the unpruned control (97.1 vs 97.9), collapsing only at K=8 (92.0). So the wrist view tolerated aggressive pruning there.

LIBERO is simulation with a different camera rig and task, so SO-101 may differ, and that is exactly what this sweep measures. But do not expect top-only pruning to be safer than uniform pruning on the strength of intuition alone.

Note also the ceiling: leaving one camera intact caps the token reduction at half of what uniform pruning reaches. With num_state_tokens=256 plus the prompt in the sequence, this variant's 30.1% image-token cut is a smaller fraction of the total sequence than it looks.

To attribute a result, compare against uniform pruning at a similar total token count, not only against the unpruned baseline.

Config

use_grid_token_sampler        : false -> true
grid_token_sampler_num_tokens :       -> [78, 0]     # [top, wrist]; 0 = not pruned
action_mode                   : continuous -> both
setup_type                    : "single SO-101 5-DOF robotic arm with gripper"

The per-camera list is honoured in three places that must agree, or the image tokens scatter into the language sequence misaligned — silently, with no error: the model builds one sampler per pruned camera, the processor emits that many image placeholder tokens per camera in the same order, and the sequence-length budget sums the per-camera counts. All three were asserted equal (274) before this checkpoint was exported.

chunk_size / n_action_steps stay at 30: apply_norm_tag_metadata() pins both from the so100_so101_molmoact2 norm tag after CLI overrides, so passing anything else has no effect.

Training data

Merged SO-101 corpus, 177,661 frames from 11 dataset repos (5 cm teleop 21.6%, 5 cm DAgger 11.2%, 2 cm teleop 62.1%, 2 cm DAgger 5.0%). Batches are drawn by uniform shuffling by default; a balanced sampler pinning a 6:2:3:1 per-batch mix is opt-in via CATEGORY_META. Normalizer statistics come from the whole corpus and are unaffected by the sampling mode.

Status of verification

Verified at construction: config and processor agree on [78, 0], exactly one sampler exists (for the top camera), scout_mlp.2.weight is (156, 512), the parameter count matches the analytic prediction, and an unpruned camera passes its grid through unchanged.

Not smoke-tested through training steps — no training was run for this variant.

Reproducing

# repo: nota-github/xpu-lerobot, branch feat/so101-train
EXPORT_INIT=1 INIT_EXPORT_DIR=/tmp/toponly-p60 GRID_NUM_TOKENS="[78,0]" HF_TOKEN=... \
  scripts/train_molmoact2_so101_grid_sampler.sh

# training run it seeds
GRID_NUM_TOKENS="[78,0]" HF_TOKEN=... WANDB_API_KEY=... \
  scripts/train_molmoact2_so101_grid_sampler.sh

References

• Grid Sampler (GridS) active token sampling, ICML 2026
• Fang et al. MolmoAct2: Action Reasoning Models for Real-world Deployment. arXiv:2605.02881