Improve model card with metadata, paper links and usage
Browse filesHi! I'm Niels, part of the community team at Hugging Face. I'm opening this PR to improve the model card for RealWonder. The updates include:
- Adding the `image-to-video` pipeline tag to the YAML metadata.
- Including links to the paper, project page, and GitHub repository.
- Adding a sample usage section for offline inference based on the repository's instructions.
- Adding the BibTeX citation.
README.md
CHANGED
|
@@ -1 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# RealWonder: Real-Time Physical Action-Conditioned Video Generation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
pipeline_tag: image-to-video
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
# RealWonder: Real-Time Physical Action-Conditioned Video Generation
|
| 6 |
+
|
| 7 |
+
RealWonder is the first real-time system for action-conditioned video generation from a single image. It uses physics simulation as an intermediate bridge to translate 3D actions into visual representations (optical flow and RGB) that a distilled video generator can process efficiently.
|
| 8 |
+
|
| 9 |
+
[**Project Page**](https://liuwei283.github.io/RealWonder/) | [**Paper (arXiv)**](https://arxiv.org/abs/2603.05449) | [**GitHub Repository**](https://github.com/liuwei283/RealWonder)
|
| 10 |
+
|
| 11 |
+
## About RealWonder
|
| 12 |
+
|
| 13 |
+
Current video generation models often lack a structural understanding of how actions affect 3D scenes. RealWonder addresses this by integrating three components:
|
| 14 |
+
1. **3D reconstruction** from single images.
|
| 15 |
+
2. **Physics simulation** for structural grounding.
|
| 16 |
+
3. **Distilled video generator** requiring only 4 diffusion steps.
|
| 17 |
+
|
| 18 |
+
The system achieves 13.2 FPS at 480x832 resolution, enabling interactive exploration of forces, robot actions, and camera controls on various materials.
|
| 19 |
+
|
| 20 |
+
## Usage
|
| 21 |
+
|
| 22 |
+
### Offline Inference
|
| 23 |
+
|
| 24 |
+
To generate a video from a single image and action-conditioned simulation results, follow these steps as described in the official implementation:
|
| 25 |
+
|
| 26 |
+
1. **Run physics simulation**:
|
| 27 |
+
```bash
|
| 28 |
+
python case_simulation.py --config_path demo_data/lamp/config.yaml
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
2. **Run video generation**:
|
| 32 |
+
```bash
|
| 33 |
+
python infer_sim.py \
|
| 34 |
+
--checkpoint_path ckpts/Realwonder-Distilled-AR-I2V-Flow/sink_size=1-attn_size=21-frame_per_block=3-denoising_steps=4/step=000800.pt \
|
| 35 |
+
--sim_data_path result/lamp/final_sim \
|
| 36 |
+
--output_path result/lamp/final_sim/final.mp4
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
## Citation
|
| 40 |
+
|
| 41 |
+
```bibtex
|
| 42 |
+
@misc{realwonder2026,
|
| 43 |
+
title={RealWonder: Real-Time Physical Action-Conditioned Video Generation},
|
| 44 |
+
author={Liu, Wei and Chen, Ziyu and Li, Zizhang and Wang, Yue and Yu, Hong-Xing and Wu, Jiajun},
|
| 45 |
+
year={2026},
|
| 46 |
+
eprint={2603.05449},
|
| 47 |
+
archivePrefix={arXiv},
|
| 48 |
+
primaryClass={cs.CV},
|
| 49 |
+
url={https://arxiv.org/abs/2603.05449},
|
| 50 |
+
}
|
| 51 |
+
```
|