|
|
| --- |
| license: other |
| license_name: amuse-research-license |
| license_link: https://github.com/kiranchhatre/amuse/blob/main/LICENSE |
| extra_gated_heading: Request access to the AMUSE research artifacts |
| extra_gated_description: Access is manually reviewed. For questions, contact amuse@tue.mpg.de or chhatre@kth.se. |
| extra_gated_button_content: Request access |
| extra_gated_prompt: By requesting access you agree to use these artifacts only under the AMUSE non-commercial research license. |
| extra_gated_fields: |
| Affiliation: text |
| Intended use: text |
| I agree to the AMUSE license terms: checkbox |
| tags: |
| - speech-driven-gesture-generation |
| - 3d-motion |
| - emotion |
| - smplx |
| library_name: pytorch |
| datasets: |
| - kiranchhatre/amuse-beat |
| --- |
| |
| # AMUSE LPDM |
|
|
| Paper page: [Hugging Face Papers 2312.04466](https://huggingface.co/papers/2312.04466) |
| ArXiv: [2312.04466](https://arxiv.org/abs/2312.04466) |
|
|
| This gated model repo contains the released AMUSE LPDM checkpoint folder `LPDM_20231028-210758_actors_smplx` |
| plus a lightweight NPZ-only inference wrapper. |
|
|
| ## What is included |
|
|
| - `saved-models/LPDM_20231028-210758_actors_smplx/` |
| - `configs/`, `dm/`, and `models/` required for inference |
| - `infer_npz.py` |
| - `configs/release_base.json` |
|
|
| ## What is not included |
|
|
| - SMPL-X model files |
| - Blender resources and Blender add-ons |
| - Purchased third-party assets |
|
|
| SMPL-X and Blender are not needed to generate motion NPZs. The released inference path outputs: |
|
|
| - `poses`: axis-angle joint rotations with shape `[T, 55, 3]` |
| - `trans`: root translation with shape `[T, 3]` |
| - `mocap_frame_rate`: `30.0` |
|
|
| ## Quick start |
|
|
| ```bash |
| python infer_npz.py \ |
| --audio /path/to/input.wav \ |
| --output /path/to/output_motion.npz |
| ``` |
|
|
| The script uses the local LPDM checkpoint in this repo and will fetch the gated audio checkpoint |
| from `kiranchhatre/amuse-audio` automatically if it is not already available locally. |
|
|
| ## Notes |
|
|
| - The released model is tuned for 10-second chunks at 16 kHz audio. |
| - Longer audio is processed chunk-by-chunk and concatenated in time. |
| - No Blender or SMPL-X downloads are required for this NPZ-only path. |
|
|
| ## Citation |
|
|
| If you use these artifacts, please cite AMUSE: |
|
|
|
|
| ```bibtex |
| @InProceedings{Chhatre_2024_CVPR, |
| author = {Chhatre, Kiran and Daněček, Radek and Athanasiou, Nikos and Becherini, Giorgio and Peters, Christopher and Black, Michael J. and Bolkart, Timo}, |
| title = {{AMUSE}: Emotional Speech-driven {3D} Body Animation via Disentangled Latent Diffusion}, |
| booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, |
| month = {June}, |
| year = {2024}, |
| pages = {1942-1953}, |
| url = {https://amuse.is.tue.mpg.de}, |
| } |
| ``` |
|
|
|
|
