Duplicate from facebook/audiobox-aesthetics

Co-authored-by: Andros Tjandra <androstj@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+aes_model.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,157 @@
+---
+license: cc-by-4.0
+pipeline_tag: audio-classification
+tags:
+- model_hub_mixin
+- pytorch_model_hub_mixin
+---
+
+This model has been pushed to the Hub using the [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) integration:
+- Code: https://github.com/facebookresearch/audiobox-aesthetics
+- Paper: https://huggingface.co/papers/2502.05139
+
+--- README below copied from https://github.com/facebookresearch/audiobox-aesthetics
+
+# audiobox-aesthetics
+
+[![PyPI - Version](https://img.shields.io/pypi/v/audiobox-aesthetics)](https://pypi.org/project/audiobox-aesthetics/) [![Hugging Face Model](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue)](https://huggingface.co/facebook/audiobox-aesthetics)
+
+Unified automatic quality assessment for speech, music, and sound.
+
+* Paper [arXiv](https://arxiv.org/abs/2502.05139) / [MetaAI](https://ai.meta.com/research/publications/meta-audiobox-aesthetics-unified-automatic-quality-assessment-for-speech-music-and-sound/).
+* Blogpost [ai.meta.com](https://ai.meta.com/blog/machine-intelligence-research-new-models/)
+
+<img src="assets/aes_model.png" alt="Model" height="400px">
+
+## Installation
+
+1. Install via pip
+ ```
+ pip install audiobox_aesthetics
+ ```
+
+2. Install directly from source
+
+ This repository requires Python 3.9 and Pytorch 2.2 or greater. To install, you can clone this repo and run:
+ ```
+ pip install -e .
+ ```
+
+## Pre-trained Models
+
+Model | S3 | HuggingFace
+|---|---|---|
+All axes | [checkpoint.pt](https://dl.fbaipublicfiles.com/audiobox-aesthetics/checkpoint.pt) | [HF Repo](https://huggingface.co/facebook/audiobox-aesthetics)
+
+## Usage
+
+### How to run prediction using CLI:
+
+1. Create a jsonl files with the following format
+ ```
+ {"path":"/path/to/a.wav"}
+ {"path":"/path/to/b.flac"}
+ ...
+ {"path":"/path/to/z.wav"}
+ ```
+ or if you only want to predict aesthetic scores from certain timestamp
+ ```
+ {"path":"/path/to/a.wav", "start_time":0, "end_time": 5}
+ {"path":"/path/to/b.flac", "start_time":3, "end_time": 10}
+ ```
+ and save it as `input.jsonl`
+
+2. Run following command
+ ```
+ audio-aes input.jsonl --batch-size 100 > output.jsonl
+ ```
+ If you haven't downloade the checkpoint, the script will try to download it automatically. Otherwise, you can provide the path by `--ckpt /path/to/checkpoint.pt`
+
+ If you have SLURM, run the following command
+ ```
+ audio-aes input.jsonl --batch-size 100 --remote --array 5 --job-dir $HOME/slurm_logs/ --chunk 1000 > output.jsonl
+ ```
+ Please adjust CPU & GPU settings using `--slurm-gpu, --slurm-cpu` depending on your nodes.
+
+
+3. Output file will contain the same number of rows as `input.jsonl`. Each row contains 4 axes of prediction with a JSON-formatted dictionary. Check the following table for more info:
+ 
+ Axes name | Full name
+ |---|---|
+ CE | Content Enjoyment
+ CU | Content Usefulness
+ PC | Production Complexity
+ PQ | Production Quality
+    
+ Output line example:
+ ```
+ {"CE": 5.146, "CU": 5.779, "PC": 2.148, "PQ": 7.220}
+ ```
+
+4. (Extra) If you want to extract only one axis (i.e. CE), post-process the output file with the following command using `jq` utility: 
+    
+    ```jq '.CE' output.jsonl > output-aes_ce.txt```
+
+
+### How to run prediction from Python script or interpreter
+
+1. Infer from file path
+```
+from audiobox_aesthetics.infer import initialize_predictor
+predictor = initialize_predictor()
+predictor.forward([{"path":"/path/to/a.wav"}, {"path":"/path/to/b.flac"}])
+```
+
+2. Infer from torch tensor
+```
+from audiobox_aesthetics.infer import initialize_predictor
+predictor = initialize_predictor()
+wav, sr = torchaudio.load("/path/to/a.wav")
+predictor.forward([{"path":wav, "sample_rate": sr}])
+```
+
+## Evaluation dataset
+We released our evaluation dataset consisting of 4 axes of aesthetic annotation scores. 
+
+Here, we show an example of how to read and re-map each annotation to the actual audio file.
+```
+{
+ "data_path": "/your_path/LibriTTS/train-clean-100/1363/139304/1363_139304_000011_000000.wav", 
+ "Production_Quality": [8.0, 8.0, 8.0, 8.0, 8.0, 9.0, 8.0, 5.0, 8.0, 8.0], 
+ "Production_Complexity": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], 
+ "Content_Enjoyment": [8.0, 6.0, 8.0, 5.0, 8.0, 8.0, 8.0, 6.0, 8.0, 6.0], 
+ "Content_Usefulness": [8.0, 6.0, 8.0, 7.0, 8.0, 9.0, 8.0, 6.0, 10.0, 7.0]
+}
+```
+1. Recognize the dataset name from data_path. In the example, it is LibriTTS.
+2. Replace "/your_path/" into your downloaded LibriTTS directory. 
+3. Each axis contains 10 scores annotated by 10 different human annotators.
+
+data_path | URL
+|---|---|
+LibriTTS |  https://openslr.org/60/
+cv-corpus-13.0-2023-03-09 | https://commonvoice.mozilla.org/en/datasets
+EARS | https://sp-uhh.github.io/ears_dataset/
+MUSDB18 | https://sigsep.github.io/datasets/musdb.html
+musiccaps | https://www.kaggle.com/datasets/googleai/musiccaps
+(audioset) unbalanced_train_segments | https://research.google.com/audioset/dataset/index.html 
+PAM | https://zenodo.org/records/10737388
+
+## License
+The majority of audiobox-aesthetics is licensed under CC-BY 4.0, as found in the LICENSE file.
+However, portions of the project are available under separate license terms: [https://github.com/microsoft/unilm](https://github.com/microsoft/unilm) is licensed under MIT license.
+
+## Citation
+If you found this repository useful, please cite the following BibTeX entry.
+
+```
+@article{tjandra2025aes,
+    title={Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound},
+    author={Andros Tjandra and Yi-Chiao Wu and Baishan Guo and John Hoffman and Brian Ellis and Apoorv Vyas and Bowen Shi and Sanyuan Chen and Matt Le and Nick Zacharov and Carleigh Wood and Ann Lee and Wei-Ning Hsu},
+    year={2025},
+    url={https://arxiv.org/abs/2502.05139}
+}
+```
+
+## Acknowledgements
+Part of the model code is copied from [https://github.com/microsoft/unilm/tree/master/wavlm](WavLM).

checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4931a7a01c3e6733352e9d85371835f03bf9135f8b31e1583c23538811d4a32
+size 415520834

config.json

@@ -0,0 +1,29 @@
+{
+  "normalize_embed": true,
+  "nth_layer": 13,
+  "output_dim": 1,
+  "precision": "32",
+  "proj_act_fn": "gelu",
+  "proj_dropout": 0.0,
+  "proj_ln": true,
+  "proj_num_layer": 5,
+  "target_transform": {
+    "CE": {
+      "mean": 5.06865,
+      "std": 1.93029
+    },
+    "CU": {
+      "mean": 5.73633,
+      "std": 1.75669
+    },
+    "PC": {
+      "mean": 3.18591,
+      "std": 1.86637
+    },
+    "PQ": {
+      "mean": 6.57505,
+      "std": 1.51466
+    }
+  },
+  "use_weighted_layer_sum": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5a3c2412649cc2384ec525ffd5180ce6c4778f43bed6108e0a1303de04d014e
+size 415472992