---
license: apache-2.0
language:
- en
pipeline_tag: feature-extraction
tags:
- audio-retrieval
- embedding
- custom_code
- multimodal
base_model: mispeech/midashenglm-7b-0804-fp32
---
ALM2Vec-FT
**ALM2Vec** is a universal audio embedding model for retrieval, derived from a pretrained large audio–language model (LALM). Instead of being optimized only for audio–caption matching like conventional contrastive dual-encoders, it transfers the audio understanding, instruction-following, and reasoning abilities of LALMs into a single unified embedding space that works across audio domains, task types, and user intents.
Its key feature is **instruction-aware retrieval**: a natural-language instruction guides the embedding, so the *same* audio can be encoded differently for different needs. This supports:
- **Instruction-aware retrieval** — focus the embedding on a specific aspect of the audio.
- **Text ↔ audio retrieval** — bidirectional matching between audio and text.
- **Audio question answering** — match an audio query plus a question against candidate answers.
ALM2Vec achieves competitive results on standard audio and speech retrieval benchmarks while adding these controllable retrieval capabilities. See the [project page](https://caml-labs.github.io/ALM2Vec/) for interactive demos.
This repository hosts the **finetune** checkpoint, built on [MiDashengLM](https://huggingface.co/mispeech/midashenglm-7b-0804-fp32).
Requirements: `transformers>=4.52`, `torch`, `safetensors`, and `torchaudio` for non-WAV audio. Requires a GPU (~31GB weights) and `trust_remote_code=True`.
## Example
```python
import torch
from transformers import AutoModel, AutoTokenizer
repo_id = "cara-ai/ALM2Vec-FT"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
repo_id, trust_remote_code=True, torch_dtype=torch.float32
).cuda().eval()
model.set_tokenizer(tokenizer)
query_text = ["q1", "q2"]
query_audio = ["/path/to/audio1.wav", "https://example.com/audio2.wav"]
doc_text = ["d1", "d2"]
query_emb = model.encode_query(text=query_text, audio=query_audio, task="query")
doc_emb = model.encode_document(text=doc_text, task="document")
similarity = query_emb @ doc_emb.T
print(similarity)
```
## Results
**ALM2Vec-FT** is the checkpoint hosted in this repository; **ALM2Vec-PT** is the pretrain variant. In every table, **bold** marks the best score and underline the second best.
### Text–audio retrieval — AudioCaps
| Method | T→A R@1 | T→A R@5 | T→A R@10 | A→T R@1 | A→T R@5 | A→T R@10 |
| --- | --- | --- | --- | --- | --- | --- |
| LAION-CLAP | 36.1 | 71.8 | 83.9 | 46.8 | 82.9 | 90.7 |
| MS-CLAP | 15.4 | 47.2 | 64.5 | 32.0 | 66.0 | 79.2 |
| WavCaps-CLAP-PT | 39.7 | 74.5 | 86.1 | 51.7 | 82.3 | 90.6 |
| WavCaps-CLAP-FT | 42.2 | 76.5 | 87.1 | 54.6 | **85.2** | **92.4** |
| JINA-Embed.-v5 | 20.4 | 50.3 | 64.4 | 23.1 | 52.7 | 67.2 |
| **ALM2Vec-PT** | 40.0 | 74.5 | 85.9 | 43.8 | 74.3 | 86.5 |
| **ALM2Vec-FT** | **43.2** | **78.0** | **87.8** | **55.5** | 80.0 | 88.2 |
### Text–audio retrieval — Clotho
| Method | T→A R@1 | T→A R@5 | T→A R@10 | A→T R@1 | A→T R@5 | A→T R@10 |
| --- | --- | --- | --- | --- | --- | --- |
| LAION-CLAP | 16.1 | 38.3 | 51.1 | 22.7 | 48.5 | 60.8 |
| MS-CLAP | 15.6 | 38.9 | 51.4 | 22.1 | 48.9 | 62.0 |
| WavCaps-CLAP-PT | 19.5 | 45.2 | 58.2 | 23.4 | 50.9 | 63.4 |
| WavCaps-CLAP-FT | 19.7 | 45.7 | 59.4 | 26.9 | 52.6 | 64.9 |
| JINA-Embed.-v5 | 9.2 | 23.9 | 35.0 | 10.5 | 24.7 | 34.3 |
| **ALM2Vec-PT** | 19.2 | 43.4 | 55.7 | 17.9 | 39.4 | 52.2 |
| **ALM2Vec-FT** | **24.8** | **52.9** | **65.8** | **27.9** | **52.7** | **66.3** |
### Speech retrieval — LibriSQA
| Method | T→S R@1 | T→S R@5 | T→S R@10 | S→T R@1 | S→T R@5 | S→T R@10 |
| -------------- | --------------- | -------- | -------- | --------------- | -------- | -------- |
| LAION-CLAP † | 0.0 | 0.1 | 0.8 | 0.1 | 0.2 | 0.6 |
| Whisper+BGE | 83.7 | 93.3 | 94.9 | 85.2 | 93.4 | 95.3 |
| CLSR | **85.0** | 93.4 | 95.0 | 85.5 | 94.0 | 95.6 |
| **ALM2Vec-PT** | 43.7 | 64.5 | 72.8 | 11.2 | 24.9 | 34.1 |
| **ALM2Vec-FT** | 84.7 | **94.1** | **95.8** | **86.0** | **95.2** | **97.2** |
### Audio understanding — MMAU-mini (accuracy)
| Method | Overall | Music | Sound | Speech |
| ------------------ | -------- | -------- | -------- | -------- |
| GPT-4o Audio ‡ | 60.8 | 63.2 | 64.6 | 56.3 |
| Gemini 2.5 Pro ‡ | 71.6 | 75.1 | 71.5 | 68.3 |
| Qwen2.5-Omni ‡ | 71.5 | 65.9 | 78.1 | 70.6 |
| Audio Flamingo 3 ‡ | **73.1** | **76.9** | 66.1 | **73.9** |
| **ALM2Vec-PT** | 66.3 | 62.3 | **78.7** | 58.0 |
| **ALM2Vec-FT** | 63.0 | 61.7 | 74.8 | 52.6 |
† LAION-CLAP is not trained for speech and effectively fails on LibriSQA; shown for reference.
‡ Generative large audio–language models, listed as reference upper bounds rather than directly comparable retrieval baselines.
## Citation
If you find this work useful, please consider citing:
```
@article{ALM2Vec2026,
title={ALM2Vec: Learning Audio Embeddings for Universal
Audio Retrieval with Large Audio-Language Models},
author={TBD},
journal={arXiv preprint arXiv:TBD},
year={2026}
}
```
## Acknowledgement
ALM2Vec is built on [MiDashengLM](https://github.com/xiaomi-research/dasheng-lm) and further trained for universal audio retrieval. We thank MiDashengLM and its underlying [Dasheng](https://github.com/RicherMans/Dasheng) audio encoder for their open-source contributions.