README.md

---
library_name: lerobot
tags:
- Robotics
- LeRobot
- Safetensors
- vision-language-action
- imitation-learning
- fastwam
- robotwin
- wan2.2
language:
- en
---

# FastWAM RoboTwin 3-Camera 384 (LeRobot)

FastWAM is a Vision-Language-Action policy built on Wan2.2 video-generation components and an
action diffusion expert. It predicts continuous robot action chunks from visual observations,
proprioception, and language/task context.

This checkpoint is converted to the Hugging Face / LeRobot `fastwam` policy format and is intended
for RoboTwin-style manipulation evaluation and fine-tuning.

## Model description

- Policy type: `fastwam`
- Backbone family: `Wan-AI/Wan2.2-TI2V-5B`
- Inputs: concatenated multi-view RGB image, robot state/proprioception, task context
- Outputs: continuous robot actions
- Training objective: FastWAM video/action diffusion loss
- Action representation: continuous action chunks
- Intended use: evaluation or fine-tuning on RoboTwin-style manipulation tasks
- Image feature: `observation.images.image`
- Image shape: `(3, 384, 320)`
- State shape: `(14,)`
- Action shape: `(14,)`
- Action horizon: `32`
- Number of video frames: `33`
- Torch dtype: `bfloat16`

## Quick start

### Installation

Install LeRobot from a version that includes the `fastwam` policy:

```bash
pip install "lerobot[fastwam]@git+https://github.com/huggingface/lerobot.git"
```

For full installation details, see the official LeRobot documentation:
https://huggingface.co/docs/lerobot/installation

### Load model and run `select_action`

```python
import torch

from lerobot.policies.fastwam.modeling_fastwam import FastWAMPolicy

model_id = "<namespace>/fastwam-robotwin-uncond-3cam384"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

policy = FastWAMPolicy.from_pretrained(model_id, strict=False).to(device).eval()

batch = {
    "observation.images.image": torch.zeros(1, 3, 384, 320, device=device),
    "observation.state": torch.zeros(1, 14, device=device),
    "prompt": "complete the manipulation task",
}

with torch.inference_mode():
    action = policy.select_action(batch)

print(action.shape)
```

`FastWAMPolicy.from_pretrained(...)` loads the policy weights and the local Wan sidecar components
from this same repository snapshot. It should not download the Wan2.2 backbone separately.

## Training step

For training or fine-tuning, call `forward(...)` and use the returned `loss` key:

```python
policy.train()

outputs = policy.forward(batch)
loss = outputs["loss"]
loss.backward()
```

The training batch must contain FastWAM-ready tensors such as `video`, `action`, `context`, and
`context_mask`, or LeRobot observation/action keys that can be adapted by the policy wrapper.

## Fine-tuning

A typical fine-tuning command follows the standard LeRobot training flow:

```bash
lerobot-train \
  --dataset.repo_id=<your-robotwin-style-dataset> \
  --output_dir=./outputs/fastwam_finetune \
  --job_name=fastwam_finetune \
  --policy.type=fastwam \
  --policy.path=<namespace>/fastwam-robotwin-uncond-3cam384 \
  --policy.device=cuda \
  --steps=100000 \
  --batch_size=1
```

Adjust batch size and sequence settings for available GPU memory.

## Evaluate in simulation

For RoboTwin evaluation, use your RoboTwin evaluation setup and pass this repository id as the
policy path:

```bash
python scripts/robotwin/eval_robotwin_fastwam.py \
  --policy-path <namespace>/fastwam-robotwin-uncond-3cam384 \
  --device cuda
```

## Repository files

This repository is self-contained for `FastWAMPolicy.from_pretrained(...)`:

```text
config.json
model.safetensors
policy_preprocessor.json
policy_preprocessor_step_2_normalizer_processor.safetensors
policy_postprocessor.json
policy_postprocessor_step_0_unnormalizer_processor.safetensors
Wan2.2_VAE.pth
models_t5_umt5-xxl-enc-bf16.pth
google/umt5-xxl/
robotwin_uncond_3cam_384_dataset_stats.json
```

The Wan VAE, UMT5 text encoder, and tokenizer are stored beside the FastWAM policy weights.

## Notes

This checkpoint uses only the migrated Hugging Face / LeRobot serialization format:
`config.json`, `model.safetensors`, and local Wan sidecar files. Original FastWAM `.pt`
checkpoint loading is not required.