---
license: other
license_name: pantagruel-research-license
license_link: https://huggingface.co/ina-foss/ssl-music-detection-music2vec/blob/main/LICENSE.md
extra_gated_prompt: >-
  You agree to use the model according to its
  [license](https://huggingface.co/ina-foss/ssl-music-detection-music2vec/blob/main/LICENSE.md).
extra_gated_fields:
  Research institute: text
  Institutional email address: text
  Country: country
  I agree to use this model for non-commercial research use only: checkbox
  I have read, and I agree to use the model according to its license: checkbox
pipeline_tag: audio-classification
library_name: transformers
tags:
- music
- music detection
base_model:
- m-a-p/music2vec-v1
---

# Music Detection using SSL features

This model is trained to detect and segment music on an given audio file.
It uses SSL features from the [`m-a-p/music2vec-v1`](https://huggingface.co/m-a-p/music2vec-v1) model.

You can find the full list of **other music detection models** using SSL features [**here**](https://huggingface.co/collections/ina-foss/ssl-music-detection).
Their global results on Mirex, OpenBMAT and Seyerlehner are the following (see the paper for more details):

| **Link to model** | **Global F1 Score** |
|-------------------|--------------------|
| ssl-music-detection-music2vec | **91.2** |
| [ssl-music-detection-base](https://huggingface.co/ina-foss/ssl-music-detection-base) | 89.4 |
| [ssl-music-detection-no_music](https://huggingface.co/ina-foss/ssl-music-detection-no_music) | 87.1 |
| [ssl-music-detection-only_speech](https://huggingface.co/ina-foss/ssl-music-detection-only_speech) | 87.5 |
| [ssl-music-detection-only_fr](https://huggingface.co/ina-foss/ssl-music-detection-only_fr) | 87.7 |
| [ssl-music-detection-gender](https://huggingface.co/ina-foss/ssl-music-detection-gender) | 88.3 |

*Voice Activity Detection (VAD) models using SSL features can be found [here](https://huggingface.co/collections/ina-foss/ssl-vad).*

## Architecture
The model first extract features from the CNN and the first transformer layer of the [`m-a-p/music2vec-v1`](https://huggingface.co/m-a-p/music2vec-v1) SSL encoder.
Then, these features are given to a downstream model MLP, which has been trained to binary predict music for each frame.
During inference, the decoding uses a Viterbi decoder (from [Librosa](https://librosa.org/doc/latest/generated/librosa.sequence.viterbi_binary.html)).

## Data and training
It has been trained on training subsets of the following datasets :
- The *Open Broadcast Media Audio from TV* dataset ([OpenBMAT](https://www.upf.edu/web/mtg/openbmat)) by Meléndez-Catalán et al. (2019).
- The Music/Speech dataset from the [Mirex 2015](https://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection#Dataset_2) challenge.
- The Music/Speech dataset from [Seyerlehner et al., (2007)](https://www.cp.jku.at/research/papers/Seyerlehner_etal_DAFx_2007.pdf).

For detailed information about training and results associated with this model, please refer to our publication.
The training hyperparameters, original checkpoint and Tensorboard event files are available in the [training directory](https://huggingface.co/ina-foss/ssl-music-detection-music2vec/tree/main/training).

## Usage

To use this model, you need the packages listed inside the [requirements.txt file](https://huggingface.co/ina-foss/ssl-music-detection-music2vec/blob/main/requirements.txt).
Then:

```python
import librosa
from transformers import AutoModel

# loading the audio file, need to be sampled at 16kHz
audio, sr = librosa.load('/path/to/your/audio/file.wav', sr=16000)

# loading the music detection model
model = AutoModel.from_pretrained(
    'ina-foss/ssl-music-detection-music2vec',
    trust_remote_code=True
)

# running the inference
output = model(
    audio=audio,
    sampling_rate=sr
)

print(output)
```

```
[{'start': 0.0, 'stop': 56.58943157192866, 'label': False},
 {'start': 56.58943157192866, 'stop': 60.45007501250208, 'label': True},
 {'start': 60.45007501250208, 'stop': 62.870478413068845, 'label': False},
[...]
 {'start': 117.03950658443074, 'stop': 119.21986997832973, 'label': True},
 {'start': 119.21986997832973, 'stop': 119.97999666611102, 'label': False}]
```

## License and citation

The model is distributed using the [pantagruel-research-license](https://huggingface.co/ina-foss/ssl-music-detection-music2vec/blob/main/LICENSE.md).

If you use this model or find it useful in your research, publications, or applications, please cite the following work:
```
@inproceedings{pelloin2026lrec,
  author =       "Pelloin, Valentin and Bekkali, Lina and Dehak, Reda and Doukhan, David",
  year =         "2026",
  title =        "Data Selection Effects on Self-Supervised Learning of Audio Representations for French Audiovisual Broadcasts",
  booktitle={Fifteenth International Conference on Language Resources and Evaluation (LREC 2026)},
  address = "Palma, Mallorca, Spain",
  publisher = "European Language Resources Association",
}
```