Instructions to use zengxy0624/pusht-obstacles-lerobot-finetune with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use zengxy0624/pusht-obstacles-lerobot-finetune with LeRobot:
- Notebooks
- Google Colab
- Kaggle
Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: lerobot
|
| 4 |
+
tags:
|
| 5 |
+
- diffusion-policy
|
| 6 |
+
- pusht
|
| 7 |
+
- imitation-learning
|
| 8 |
+
- robotics
|
| 9 |
+
base_model: lerobot/diffusion_pusht
|
| 10 |
+
datasets:
|
| 11 |
+
- zengxy0624/pusht_obstacles
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# pusht-obstacles-lerobot-finetune
|
| 15 |
+
|
| 16 |
+
Diffusion Policy checkpoints finetuned from
|
| 17 |
+
[`lerobot/diffusion_pusht`](https://huggingface.co/lerobot/diffusion_pusht) on a
|
| 18 |
+
custom 101-episode PushT-with-obstacles dataset (legacy obstacle preset:
|
| 19 |
+
circle radius=15, 1-3 obstacles per episode).
|
| 20 |
+
|
| 21 |
+
Source code:
|
| 22 |
+
[github.com/Zengxy0624/xarm7_collect](https://github.com/Zengxy0624/xarm7_collect)
|
| 23 |
+
(see `push_t/` and `push_t/doc/`).
|
| 24 |
+
|
| 25 |
+
## Files
|
| 26 |
+
|
| 27 |
+
| File | Source run | Pipeline | Best success (n=50) | Best hit | Step |
|
| 28 |
+
|------|------------|----------|---------------------|----------|------|
|
| 29 |
+
| `lerobot_run3.5_best.pt` | jid2238141 | fixed | **0.62** | **0.12** | 44000 |
|
| 30 |
+
| `lerobot_run3.5_final.pt` | jid2238141 | fixed | 0.62 (final eval) | 0.12 | 50000 |
|
| 31 |
+
| `lerobot_run2_best.pt` | jid2235676 | broken (pre-fix) | 0.60 | 0.22 | 38000 |
|
| 32 |
+
|
| 33 |
+
The "fixed" pipeline applies three corrections matching LeRobot's official training
|
| 34 |
+
recipe (ImageNet image stats, padding-loss mask, `diffusers.get_scheduler('cosine')`)
|
| 35 |
+
that the original code accidentally bypassed by constructing `LeRobotDataset`
|
| 36 |
+
outside `make_dataset()`.
|
| 37 |
+
|
| 38 |
+
See [`push_t/doc/ft_jid2237070_2238141.md`](https://github.com/Zengxy0624/xarm7_collect/blob/main/push_t/doc/ft_jid2237070_2238141.md)
|
| 39 |
+
for the full eval timeline (25 evals across 50k training steps), phase analysis, and
|
| 40 |
+
A/B comparison vs the broken-pipeline run.
|
| 41 |
+
|
| 42 |
+
## When to use which file
|
| 43 |
+
|
| 44 |
+
- **`lerobot_run3.5_best.pt`** — default; best success and lowest obstacle-hit rate.
|
| 45 |
+
- **`lerobot_run3.5_final.pt`** — only for resume training. Same model performance
|
| 46 |
+
as best.pt (62% at both step 44k and step 50k); carries end-of-cosine-schedule state.
|
| 47 |
+
- **`lerobot_run2_best.pt`** — A/B reference. Trained with the buggy pre-fix pipeline,
|
| 48 |
+
useful for ablations.
|
| 49 |
+
|
| 50 |
+
## Loading
|
| 51 |
+
|
| 52 |
+
All ckpts contain numpy `stats` arrays, so PyTorch 2.6+ requires `weights_only=False`:
|
| 53 |
+
|
| 54 |
+
```python
|
| 55 |
+
import torch
|
| 56 |
+
from huggingface_hub import hf_hub_download
|
| 57 |
+
|
| 58 |
+
ckpt_path = hf_hub_download(
|
| 59 |
+
repo_id="zengxy0624/pusht-obstacles-lerobot-finetune",
|
| 60 |
+
filename="lerobot_run3.5_best.pt",
|
| 61 |
+
)
|
| 62 |
+
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
|
| 63 |
+
|
| 64 |
+
print(ckpt.keys()) # ['model', 'model_cfg', 'stats', 'success']
|
| 65 |
+
print(ckpt["success"]) # 0.62
|
| 66 |
+
print(ckpt["model_cfg"]) # full DiffusionConfig dict (ResNet-18 + GroupNorm + 96x96)
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
For full evaluation, use `push_t/test.py` from the source repo, which handles the
|
| 70 |
+
processor pipeline (ImageNet stats override, deterministic cudnn) correctly.
|
| 71 |
+
|
| 72 |
+
## Training details
|
| 73 |
+
|
| 74 |
+
- **Base model:** `lerobot/diffusion_pusht` (ImageNet-pretrained ResNet-18 backbone,
|
| 75 |
+
GroupNorm, spatial-softmax 32 keypoints, U-Net 1D, DDPM 100 timesteps)
|
| 76 |
+
- **Architecture:** image_shape=[3, 96, 96], crop_shape=[84, 84], n_obs_steps=2,
|
| 77 |
+
horizon=16, n_action_steps=8, down_dims=[512, 1024, 2048]
|
| 78 |
+
- **Hyperparameters:** bs=64, lr=3e-5, warmup_steps=100, weight_decay=1e-6,
|
| 79 |
+
grad_clip=10.0, AMP fp16
|
| 80 |
+
- **Schedule:** linear warmup -> cosine decay over 50000 steps
|
| 81 |
+
- **Eval:** 50 episodes per checkpoint, every 2000 training steps
|
| 82 |
+
- **Hardware:** 1x H100 SXM5 on NCSA Delta AI
|
| 83 |
+
|
| 84 |
+
## Limitations
|
| 85 |
+
|
| 86 |
+
- Eval n=50 gives Wilson 95% CI of approximately +/-14%. Reported success rates
|
| 87 |
+
should be treated as point estimates with this uncertainty.
|
| 88 |
+
- Trained only on the legacy circular-obstacle preset. Will not generalize to the
|
| 89 |
+
alternative `mixed` preset (rectangles, triangles, walls) without retraining.
|
| 90 |
+
- Both runs show val_loss divergence consistent with small-dataset overfitting; the
|
| 91 |
+
101-episode dataset is the bottleneck, not the training stack.
|
| 92 |
+
|
| 93 |
+
## Citation
|
| 94 |
+
|
| 95 |
+
If you use these checkpoints, please cite:
|
| 96 |
+
|
| 97 |
+
- The base Diffusion Policy paper: Cheng Chi et al. "Diffusion Policy: Visuomotor
|
| 98 |
+
Policy Learning via Action Diffusion" (RSS 2023, IJRR 2024).
|
| 99 |
+
- The LeRobot framework that produced the base ckpt and was used for these
|
| 100 |
+
experiments.
|