Feature Extraction
Transformers
Safetensors
qwen2_5_omni
text-to-audio
multimodal
text
image
video
audio
embedding
retrieval
Instructions to use ATH-MaaS/Ovis-Omni-Embedding-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ATH-MaaS/Ovis-Omni-Embedding-3B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="ATH-MaaS/Ovis-Omni-Embedding-3B")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("ATH-MaaS/Ovis-Omni-Embedding-3B") model = AutoModelForMultimodalLM.from_pretrained("ATH-MaaS/Ovis-Omni-Embedding-3B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -16,13 +16,13 @@ tags:
|
|
| 16 |
<img alt="GitHub" src="https://img.shields.io/badge/GitHub-Ovis--Omni--Embedding-3157C8?logo=github" style="display: inline-block; vertical-align: middle;"/>
|
| 17 |
</a>
|
| 18 |
|
| 19 |
-
# Ovis-
|
| 20 |
|
| 21 |
-
**Ovis-
|
| 22 |
|
| 23 |
The model is initialized from **Qwen2.5-Omni-3B**. Rather than attaching separate modality-specific embedding towers, it retains the native text tokenizer, vision encoder, audio encoder, and shared Thinker backbone. The speech-generation Talker and language-modeling head are removed, and the final-layer hidden state at the last non-padding token is used directly as the retrieval embedding.
|
| 24 |
|
| 25 |
-
Ovis-
|
| 26 |
|
| 27 |
## Model highlights
|
| 28 |
|
|
@@ -38,10 +38,10 @@ Ovis-Embedding-Omni-3B is designed for multimodal search, retrieval-augmented ge
|
|
| 38 |
## Architecture
|
| 39 |
|
| 40 |
<p align="center">
|
| 41 |
-
<img src="./ovis_embedding_model_architecture.png" alt="Architecture of Ovis-
|
| 42 |
</p>
|
| 43 |
|
| 44 |
-
*Figure 2 from the technical report. Ovis-
|
| 45 |
|
| 46 |
An input is formatted with a retrieval instruction through the native processor and chat template. Text, visual, and acoustic tokens are then processed as one interleaved sequence by the shared causal Transformer. Qwen2.5-Omni's time-aligned multimodal rotary position embedding preserves temporal alignment between audio and video.
|
| 47 |
|
|
@@ -67,7 +67,7 @@ Training follows three stages:
|
|
| 67 |
|
| 68 |
## Retrieval interface
|
| 69 |
|
| 70 |
-
Ovis-
|
| 71 |
|
| 72 |
1. Pair the query with the task instruction and format it with the model's native processor and chat template.
|
| 73 |
2. Encode queries and candidates independently.
|
|
@@ -81,9 +81,9 @@ No answer generation or query-candidate cross-attention is used during retrieval
|
|
| 81 |
|
| 82 |
### MMEB-v3
|
| 83 |
|
| 84 |
-
MMEB-v3 is an omni-modal benchmark comprising **190 datasets** across image, video, visual-document, text, audio, and agent retrieval. Ovis-
|
| 85 |
|
| 86 |
-
| Group | Ovis-
|
| 87 |
|---|---:|---:|---:|
|
| 88 |
| Image | **77.55** | 73.83 | +3.72 |
|
| 89 |
| Video | **64.99** | 59.37 | +5.62 |
|
|
@@ -93,17 +93,17 @@ MMEB-v3 is an omni-modal benchmark comprising **190 datasets** across image, vid
|
|
| 93 |
| Agent | **45.52** | 39.42 | +6.10 |
|
| 94 |
| **All 190 datasets** | **58.46** | 53.27 | +5.19 |
|
| 95 |
|
| 96 |
-
Across the 31 aggregate and sub-task entries in the complete comparison, Ovis-
|
| 97 |
|
| 98 |
<p align="center">
|
| 99 |
-
<img src="./ovis_blog_table1.png" alt="Complete MMEB-v3 comparison for Ovis-
|
| 100 |
</p>
|
| 101 |
|
| 102 |
*Scores are percentages and higher is better. Red marks the best result in each row, underlining marks the second best, and Ovis scores are bold. The overall score is the unweighted average over all 190 MMEB-v3 datasets. MMEB-v3 primarily uses Hit@1 for image, video, audio, and agent tasks and nDCG@5 for text and visual-document retrieval.*
|
| 103 |
|
| 104 |
### Additional benchmark results
|
| 105 |
|
| 106 |
-
| Benchmark | Ovis-
|
| 107 |
|---|---:|---|
|
| 108 |
| MAEB (beta) | **57.29** | Mean over 30 audio embedding tasks |
|
| 109 |
| MVEB (beta) | **61.77** | Mean over 23 video and audio-video embedding tasks |
|
|
@@ -145,4 +145,4 @@ This model is released under the Apache 2.0 license.
|
|
| 145 |
- [Ovis-Omni-Embedding GitHub repository](https://github.com/ATH-MaaS/Ovis-Omni-Embedding)
|
| 146 |
- [Ovis-VL-Embedding GitHub repository](https://github.com/ATH-MaaS/Ovis-VL-Embedding)
|
| 147 |
|
| 148 |
-
This repository contains the weights for **Ovis-
|
|
|
|
| 16 |
<img alt="GitHub" src="https://img.shields.io/badge/GitHub-Ovis--Omni--Embedding-3157C8?logo=github" style="display: inline-block; vertical-align: middle;"/>
|
| 17 |
</a>
|
| 18 |
|
| 19 |
+
# Ovis-Omni-Embedding-3B
|
| 20 |
|
| 21 |
+
**Ovis-Omni-Embedding-3B** is a 3B-parameter universal embedding model for text, images, visual documents, video, audio, and interleaved multimodal inputs. It maps every supported input type into one coherent representation space, enabling any-to-any retrieval with a single encoder.
|
| 22 |
|
| 23 |
The model is initialized from **Qwen2.5-Omni-3B**. Rather than attaching separate modality-specific embedding towers, it retains the native text tokenizer, vision encoder, audio encoder, and shared Thinker backbone. The speech-generation Talker and language-modeling head are removed, and the final-layer hidden state at the last non-padding token is used directly as the retrieval embedding.
|
| 24 |
|
| 25 |
+
Ovis-Omni-Embedding-3B is designed for multimodal search, retrieval-augmented generation, recommendation, visual-document retrieval, video and audio search, and agentic retrieval over tools, interfaces, and memory.
|
| 26 |
|
| 27 |
## Model highlights
|
| 28 |
|
|
|
|
| 38 |
## Architecture
|
| 39 |
|
| 40 |
<p align="center">
|
| 41 |
+
<img src="./ovis_embedding_model_architecture.png" alt="Architecture of Ovis-Omni-Embedding-3B and Ovis-Embedding-VL-2B/9B" width="100%"/>
|
| 42 |
</p>
|
| 43 |
|
| 44 |
+
*Figure 2 from the technical report. Ovis-Omni-Embedding-3B encodes text, visual, and audio inputs as an interleaved token sequence processed by the Qwen2.5-Omni Thinker with TMRoPE. The final-layer hidden state at the last non-padding token becomes the retrieval embedding, with no modality-specific projection heads.*
|
| 45 |
|
| 46 |
An input is formatted with a retrieval instruction through the native processor and chat template. Text, visual, and acoustic tokens are then processed as one interleaved sequence by the shared causal Transformer. Qwen2.5-Omni's time-aligned multimodal rotary position embedding preserves temporal alignment between audio and video.
|
| 47 |
|
|
|
|
| 67 |
|
| 68 |
## Retrieval interface
|
| 69 |
|
| 70 |
+
Ovis-Omni-Embedding-3B is a bi-encoder, not a cross-encoder:
|
| 71 |
|
| 72 |
1. Pair the query with the task instruction and format it with the model's native processor and chat template.
|
| 73 |
2. Encode queries and candidates independently.
|
|
|
|
| 81 |
|
| 82 |
### MMEB-v3
|
| 83 |
|
| 84 |
+
MMEB-v3 is an omni-modal benchmark comprising **190 datasets** across image, video, visual-document, text, audio, and agent retrieval. Ovis-Omni-Embedding-3B achieves **58.46 overall**, outperforming the strongest compared baseline by **5.19 points**, and ranks first on the aggregate score of every modality group.
|
| 85 |
|
| 86 |
+
| Group | Ovis-Omni-Embedding-3B | Best compared baseline | Margin |
|
| 87 |
|---|---:|---:|---:|
|
| 88 |
| Image | **77.55** | 73.83 | +3.72 |
|
| 89 |
| Video | **64.99** | 59.37 | +5.62 |
|
|
|
|
| 93 |
| Agent | **45.52** | 39.42 | +6.10 |
|
| 94 |
| **All 190 datasets** | **58.46** | 53.27 | +5.19 |
|
| 95 |
|
| 96 |
+
Across the 31 aggregate and sub-task entries in the complete comparison, Ovis-Omni-Embedding-3B ranks first on 22 and second on 8. MultiConIR is the only entry on which it falls outside the top two.
|
| 97 |
|
| 98 |
<p align="center">
|
| 99 |
+
<img src="./ovis_blog_table1.png" alt="Complete MMEB-v3 comparison for Ovis-Omni-Embedding-3B and four omni-modal embedding baselines" width="100%"/>
|
| 100 |
</p>
|
| 101 |
|
| 102 |
*Scores are percentages and higher is better. Red marks the best result in each row, underlining marks the second best, and Ovis scores are bold. The overall score is the unweighted average over all 190 MMEB-v3 datasets. MMEB-v3 primarily uses Hit@1 for image, video, audio, and agent tasks and nDCG@5 for text and visual-document retrieval.*
|
| 103 |
|
| 104 |
### Additional benchmark results
|
| 105 |
|
| 106 |
+
| Benchmark | Ovis-Omni-Embedding-3B | Evaluation scope |
|
| 107 |
|---|---:|---|
|
| 108 |
| MAEB (beta) | **57.29** | Mean over 30 audio embedding tasks |
|
| 109 |
| MVEB (beta) | **61.77** | Mean over 23 video and audio-video embedding tasks |
|
|
|
|
| 145 |
- [Ovis-Omni-Embedding GitHub repository](https://github.com/ATH-MaaS/Ovis-Omni-Embedding)
|
| 146 |
- [Ovis-VL-Embedding GitHub repository](https://github.com/ATH-MaaS/Ovis-VL-Embedding)
|
| 147 |
|
| 148 |
+
This repository contains the weights for **Ovis-Omni-Embedding-3B**.
|