haodongli666/10102026_zhenji

AGL real-robot handoff: Dynamic_Task

This repository contains 30 independently fine-tuned policies: two model families × five datasets × three training stages. Each checkpoint is the selected EMA model at epoch 10, 20, or 30. Start with epoch 30 for the main real-robot experiment; the earlier stages are retained for comparison.

Model family	Identifier	Inference entry
I-JEPA-H (Ours), H/14 ImageNet-22K encoder	i_jepa_h	inference_ijepa.py
DINOv3-H+ (Ours), H+/16 encoder	dinov3_h	inference_dinov3.py

Both entries expose the same Policy API and use the shared agl_inference/ implementation. The exact frozen visual encoders, action normalization, model configurations, and test images are included. No original RoboTwin checkpoint, training dataset download, or training-machine path is needed for inference.

1. Install and download

Access: this repository uses manual gated access. The collaborator should request access on the repository page, wait for the owner haodongli666 to approve the request, and authenticate with their own approved Hugging Face account. A read token is sufficient for downloading.

The validated stack is Python 3.10, PyTorch 2.7.1 + CUDA 12.8, Transformers 4.57.6, on an NVIDIA A100. Use a CUDA GPU supporting BF16 and a working C/C++ compiler for TorchInductor. Keep these package versions when reproducing the reference outputs.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch==2.7.1 torchvision==0.22.1 \
  --index-url https://download.pytorch.org/whl/cu128
python -m pip install huggingface_hub==0.36.2
hf auth login

# Full release: approximately 52.5 GB of weights. Choose a local SSD directory.
hf download haodongli666/10102026_zhenji --local-dir ./10102026_zhenji
cd 10102026_zhenji
python -m pip install -r requirements.txt

export TORCHINDUCTOR_CACHE_DIR="$PWD/.cache/torchinductor"
export TRITON_CACHE_DIR="$PWD/.cache/triton"
export OMP_NUM_THREADS=4

For a smaller download, replace the full hf download command with this example for I-JEPA-H / Dynamic_Cup / epoch 30. Change the family, dataset, and epoch paths together to select another model:

hf download haodongli666/10102026_zhenji --local-dir ./10102026_zhenji \
  --include "*.py" "agl_inference/**" "requirements.txt" "README.md" \
  "NOTICE.md" "licenses/*" "manifest.json" \
  "encoders/i_jepa_h/*" \
  "checkpoints/epoch_30/i_jepa_h/Dynamic_Cup/*" \
  "examples/Dynamic_Cup.npz" "examples/reference_i_jepa_h.npz"

Each policy head occupies about 1.55 GB. The shared encoder occupies about 2.52 GB for I-JEPA-H or 3.36 GB for DINOv3-H+, and is stored once per family. manifest.json and reports/model_index.csv list every model and SHA256.

2. Run an offline check before connecting the controller

The defaults enable BF16 precast + torch.compile + CUDA Graph. Compilation and graph warmup happen during construction and can take a few minutes on the first launch. Initialize the policy before starting the control loop; subsequent predictions reuse the prepared model and graph.

python validate_inference.py --family i_jepa_h \
  --dataset Dynamic_Cup --epochs 30 --report ijepa_validation.json
python validate_inference.py --family dinov3_h \
  --dataset Dynamic_Cup --epochs 30 --report dinov3_validation.json

The validator checks all three included camera triplets, the entire 32-step action chunk, exact image preprocessing, changing-image graph replay, ownership of returned outputs, and concurrent callers. It compares against full outputs generated by the original training implementation. The numerical tolerances are regression checks, not robot motion limits.

To save a predicted chunk and measure warm inference latency:

python inference_ijepa.py --dataset Dynamic_Cup --epochs 30 \
  --output ijepa_actions.npy --benchmark 50 --report ijepa_benchmark.json
python inference_dinov3.py --dataset Speed_Cup_40 --epochs 20 \
  --output dino_actions.npy --benchmark 50 --report dino_benchmark.json

These commands use a demonstration fixture by default and do not actuate hardware. For three saved live images:

python inference_dinov3.py --dataset Dynamic_Cup --epochs 30 \
  --left-eye left_eye.png --left-wrist left_wrist.png \
  --right-wrist right_wrist.png --output actions.npy

3. Integrate with the real-robot loop

import torch
from inference_ijepa import Policy
# For DINOv3-H+ instead:
# from inference_dinov3 import Policy

torch.set_num_threads(4)
policy = Policy(
    root="/path/to/10102026_zhenji",
    dataset="Dynamic_Cup",   # or Speed_Cup_20 / 30 / 40 / 50
    epochs=30,              # 10, 20, or 30
    device="cuda:0",
    bf16_precast=True,
    compile=True,
    cuda_graph=True,
)

# Each input is an HWC uint8 RGB NumPy array, normally (480, 640, 3).
actions = policy.predict_chunk(left_eye_rgb, left_wrist_rgb, right_wrist_rgb)
assert actions.shape == (32, 19)

# Route these absolute commands through the existing robot controller:
left_arm_qpos = actions[:, 0:7]
right_arm_qpos = actions[:, 7:14]
left_gripper_qpos = actions[:, 14]
right_gripper_qpos = actions[:, 15]
body_qpos = actions[:, 16:19]  # exactly zero, matching the demonstrations

The model predicts 32 absolute actions on a 30 Hz time grid. The controller chooses how many leading actions to execute before requesting a new chunk. Record that execution horizon and the measured inference latency with each trial. The 32-step prediction horizon does not require executing all 32 actions open loop.

No proprioception, previous actions, language prompt, or future image is needed. Each dataset has its own checkpoint and normalization. The policy already undoes normalization; pass its returned values through the same action mapping used to collect the demonstrations.

Cameras and preprocessing

Argument	Dataset camera	Position in the composite
left_eye_rgb	observation.images.color_0	Top
left_wrist_rgb	observation.images.color_2	Bottom left
right_wrist_rgb	observation.images.color_3	Bottom right

color_1, the right stereo eye, is excluded. Inputs must be RGB, uint8, HWC, 4:3; the demonstrations use 640×480 cameras. If the capture library returns BGR, convert to RGB first.

The implementation builds the same “品” composite as training: top image 256×320, two lower images 128×160, producing a 384×320 image (dimensions here are height × width). It normalizes to [-1,1], resizes to 224×224 with the original FP32 bilinear antialiasing, and casts to BF16. DINOv3 then applies the same internal 256×256 resize and ImageNet normalization used during training. Supply the raw camera frames to predict_chunk; preprocessing is handled inside the policy.

Action fields

Columns	Original dataset names	Meaning
0:7	left_arm.qpos.0 … .6	Seven left-arm joint positions, radians
7:14	right_arm.qpos.0 … .6	Seven right-arm joint positions, radians
14	left_ee.qpos.0	Left gripper motor position, native dataset units
15	right_ee.qpos.0	Right gripper motor position, native dataset units
16:19	body.qpos.0 … .2	Three recorded body position fields, fixed to zero

The returned array is float32 (32,19). Gripper values retain the demonstration scale, approximately 0 to 5.4, rather than a binary open/close encoding. The model learns the first 16 fields; the wrapper appends the final three zeros. No clipping, smoothing, or temporal ensembling is added.

4. Acceleration controls

Option	Default	What runs
BF16 precast	On	Frozen encoders use BF16; 66 eligible action-head linear modules are cast once before inference.
torch.compile	On	Complete action head, static batch 1; compiler preserves eager BF16 rounding.
CUDA Graph	On	Complete encoder + action-head GPU forward; new pixels are copied into stable buffers before each replay.
Condition precomputation	Off	Time/task embeddings and every AdaLN modulation are evaluated on every inference.

Fusion and the final action readout compute in FP32. Head normalization parameters, task embedding, and action-query parameters retain their original precision. Distributed checkpoint files remain the original FP32 EMA weights. The frozen encoder uses its original eager CUDA kernels inside the captured graph to preserve its trained numerical behavior. The NumPy result and low-level predict_tensor result own their storage, so a later replay does not overwrite a previous prediction.

All three controls can be switched independently. For the original eager numerical path:

python validate_inference.py --family i_jepa_h --dataset Dynamic_Cup --epochs 30 \
  --no-compile --no-cuda-graph --no-bf16-precast

For graph replay without compilation, use --no-compile. For compilation without graph replay, use --no-cuda-graph. Turning off precast leaves head weights in FP32 and uses the original BF16 autocast path; the frozen encoders still use BF16. Failures to compile or capture are reported, rather than silently changing the requested mode.

--compile-mode max-autotune-no-cudagraphs is an optional, slower startup configuration for machine-specific tuning. The default default mode is the validated release configuration. Inductor's own CUDA Graph capture is disabled; the policy owns the explicit graph. Initialize policy instances serially in a process before starting concurrent prediction callers.

Measured warm latency

These are 50 synchronized measurements on a shared A100-SXM4-80GB using Dynamic_Cup / epoch 30, all three accelerations enabled, and four CPU threads. Other GPU workloads remained running. Initialization, compilation, and capture are excluded.

Model	GPU input → normalized tensor, median / p95	RGB input → NumPy actions, median / p95
I-JEPA-H	19.4 / 23.5 ms	26.9 / 31.1 ms
DINOv3-H+	31.8 / 36.7 ms	36.4 / 41.0 ms

Both columns use synchronized wall time; the first also includes the Python wrapper and owned-output copy. The second includes preprocessing and CPU transfer, but excludes camera acquisition and robot communication. Treat these as reference measurements; benchmark the actual deployment machine and choose the action execution horizon accordingly. The first uncached initialization took about 114–115 seconds in this run.

5. Training stages and curves

All available demonstrations were used for fitting, with frozen visual encoders and independently trained action heads. The stages are cumulative: 10 epochs, then continuation to 20, then continuation to 30.

Dataset	Episodes	Frames	Training FPS
Dynamic_Cup	50	28,643	30
Speed_Cup_20	50	11,502	30
Speed_Cup_30	51	10,344	30
Speed_Cup_40	50	10,462	30
Speed_Cup_50	49	8,799	30

The values below are training episode-mean normalized active-action L1; lower is better. Dynamic_Cup uses all 16 learned action fields for this metric. Speed_Cup uses the seven right-arm fields and right gripper. These are fitting metrics; real-robot success rates are measured during deployment.

Dataset	I-JEPA 10	I-JEPA 20	I-JEPA 30	DINOv3 10	DINOv3 20	DINOv3 30
Dynamic_Cup	0.01574	0.01000	0.00852	0.01691	0.01045	0.00878
Speed_Cup_20	0.02186	0.01175	0.00937	0.02426	0.01254	0.00983
Speed_Cup_30	0.02669	0.01402	0.01104	0.02873	0.01459	0.01122
Speed_Cup_40	0.02724	0.01429	0.01122	0.03027	0.01548	0.01202
Speed_Cup_50	0.02799	0.01546	0.01239	0.03050	0.01603	0.01254

The figure is the original continuation_30ep/reports/training_curves.png from the experiment, copied without modification. Full-resolution PNG · PDF · Per-epoch raw/EMA metrics

6. Files, checks, and experiment records

checkpoints/
  epoch_10/  epoch_20/  epoch_30/
    i_jepa_h/  dinov3_h/
      Dynamic_Cup/  Speed_Cup_20/  Speed_Cup_30/  Speed_Cup_40/  Speed_Cup_50/
        model.safetensors       # selected EMA action head
        config.json             # portable architecture and encoder reference
        data_contract.json      # exact normalization and action ordering
        training_config.json    # training hyperparameters for this stage
encoders/
  i_jepa_h/  dinov3_h/           # exact shared frozen backbones
inference_ijepa.py
inference_dinov3.py
validate_inference.py
agl_inference/
examples/                       # raw camera fixtures and full reference actions
reports/                        # fitting curves, integrity and numerical checks
manifest.json
requirements.txt
NOTICE.md

The loader checks the policy and encoder SHA256 by default. reports/uploaded_weights_verified.json records the Hub hashes for all 30 policy heads + 2 encoders. reports/validation_summary.json records the release's numerical and replay checks. The individual family/mode reports contain per-checkpoint results.

All 30 models × 3 camera triplets passed in each of the three tested modes:

Mode	Comparison to archived full action chunks
Original eager path	Bitwise identical normalized outputs
BF16 precast, without compile/graph	Bitwise identical normalized outputs
BF16 precast + compiled action head + full CUDA Graph	Passed the fixed numerical, fresh-input, output-ownership, and concurrency checks

Across the 90 accelerated model/input pairs, the maximum joint-position difference was 0.00407 radians, and the maximum gripper difference was 0.01789 native units. The largest per-model mean normalized absolute difference was 0.000923. These comparisons use the original eager inference outputs as the reference.

Both family entry points were also tested from fresh authenticated Hub downloads, with Hugging Face and Transformers offline mode enabled, using Dynamic_Cup / epoch 30. The default mode and each acceleration independently disabled passed the same checks. reports/hub_roundtrip_verification.json records the tested code revision, exact source hashes, per-family reports, and latency measurements.

For each real-robot trial, record: model family, dataset, training epoch, checkpoint SHA256 (policy.checkpoint_sha256), acceleration settings (policy.runtime_info), action execution horizon, inference latency, and success/failure. Preserve the same camera geometry and action mapping used for the demonstrations.

Upstream encoder identities and component license texts are included in NOTICE.md and licenses/. The I-JEPA encoder is the ImageNet-22K H/14 release; the DINOv3 encoder is the H+/16 LVD-1689M release.