README.md

metadata

license: other
license_name: amuse-research-license
license_link: https://github.com/kiranchhatre/amuse/blob/main/LICENSE
extra_gated_heading: Request access to the AMUSE research artifacts
extra_gated_description: >-
  Access is manually reviewed. For questions, contact amuse@tue.mpg.de or
  chhatre@kth.se.
extra_gated_button_content: Request access
extra_gated_prompt: >-
  By requesting access you agree to use these artifacts only under the AMUSE
  non-commercial research license.
extra_gated_fields:
  Affiliation: text
  Intended use: text
  I agree to the AMUSE license terms: checkbox
tags:
  - speech-driven-gesture-generation
  - 3d-motion
  - emotion
  - smplx
library_name: pytorch
datasets:
  - kiranchhatre/amuse-beat

AMUSE LPDM

Paper page: Hugging Face Papers 2312.04466
ArXiv: 2312.04466

This gated model repo contains the released AMUSE LPDM checkpoint folder LPDM_20231028-210758_actors_smplx plus a lightweight NPZ-only inference wrapper.

What is included

• saved-models/LPDM_20231028-210758_actors_smplx/
• configs/, dm/, and models/ required for inference
• infer_npz.py
• configs/release_base.json

What is not included

• SMPL-X model files
• Blender resources and Blender add-ons
• Purchased third-party assets

SMPL-X and Blender are not needed to generate motion NPZs. The released inference path outputs:

• poses: axis-angle joint rotations with shape [T, 55, 3]
• trans: root translation with shape [T, 3]
• mocap_frame_rate: 30.0

Quick start

python infer_npz.py \
  --audio /path/to/input.wav \
  --output /path/to/output_motion.npz

The script uses the local LPDM checkpoint in this repo and will fetch the gated audio checkpoint from kiranchhatre/amuse-audio automatically if it is not already available locally.

Notes

• The released model is tuned for 10-second chunks at 16 kHz audio.
• Longer audio is processed chunk-by-chunk and concatenated in time.
• No Blender or SMPL-X downloads are required for this NPZ-only path.

Citation

If you use these artifacts, please cite AMUSE:

@InProceedings{Chhatre_2024_CVPR,
    author    = {Chhatre, Kiran and Daněček, Radek and Athanasiou, Nikos and Becherini, Giorgio and Peters, Christopher and Black, Michael J. and Bolkart, Timo},
    title     = {{AMUSE}: Emotional Speech-driven {3D} Body Animation via Disentangled Latent Diffusion},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {1942-1953},
    url       = {https://amuse.is.tue.mpg.de},
}