wl1982 commited on
Commit
48fcf96
·
verified ·
1 Parent(s): 62949ab

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -12
README.md CHANGED
@@ -16,13 +16,13 @@ tags:
16
  <img alt="GitHub" src="https://img.shields.io/badge/GitHub-Ovis--Omni--Embedding-3157C8?logo=github" style="display: inline-block; vertical-align: middle;"/>
17
  </a>
18
 
19
- # Ovis-Embedding-Omni-3B
20
 
21
- **Ovis-Embedding-Omni-3B** is a 3B-parameter universal embedding model for text, images, visual documents, video, audio, and interleaved multimodal inputs. It maps every supported input type into one coherent representation space, enabling any-to-any retrieval with a single encoder.
22
 
23
  The model is initialized from **Qwen2.5-Omni-3B**. Rather than attaching separate modality-specific embedding towers, it retains the native text tokenizer, vision encoder, audio encoder, and shared Thinker backbone. The speech-generation Talker and language-modeling head are removed, and the final-layer hidden state at the last non-padding token is used directly as the retrieval embedding.
24
 
25
- Ovis-Embedding-Omni-3B is designed for multimodal search, retrieval-augmented generation, recommendation, visual-document retrieval, video and audio search, and agentic retrieval over tools, interfaces, and memory.
26
 
27
  ## Model highlights
28
 
@@ -38,10 +38,10 @@ Ovis-Embedding-Omni-3B is designed for multimodal search, retrieval-augmented ge
38
  ## Architecture
39
 
40
  <p align="center">
41
- <img src="./ovis_embedding_model_architecture.png" alt="Architecture of Ovis-Embedding-Omni-3B and Ovis-Embedding-VL-2B/9B" width="100%"/>
42
  </p>
43
 
44
- *Figure 2 from the technical report. Ovis-Embedding-Omni-3B encodes text, visual, and audio inputs as an interleaved token sequence processed by the Qwen2.5-Omni Thinker with TMRoPE. The final-layer hidden state at the last non-padding token becomes the retrieval embedding, with no modality-specific projection heads.*
45
 
46
  An input is formatted with a retrieval instruction through the native processor and chat template. Text, visual, and acoustic tokens are then processed as one interleaved sequence by the shared causal Transformer. Qwen2.5-Omni's time-aligned multimodal rotary position embedding preserves temporal alignment between audio and video.
47
 
@@ -67,7 +67,7 @@ Training follows three stages:
67
 
68
  ## Retrieval interface
69
 
70
- Ovis-Embedding-Omni-3B is a bi-encoder, not a cross-encoder:
71
 
72
  1. Pair the query with the task instruction and format it with the model's native processor and chat template.
73
  2. Encode queries and candidates independently.
@@ -81,9 +81,9 @@ No answer generation or query-candidate cross-attention is used during retrieval
81
 
82
  ### MMEB-v3
83
 
84
- MMEB-v3 is an omni-modal benchmark comprising **190 datasets** across image, video, visual-document, text, audio, and agent retrieval. Ovis-Embedding-Omni-3B achieves **58.46 overall**, outperforming the strongest compared baseline by **5.19 points**, and ranks first on the aggregate score of every modality group.
85
 
86
- | Group | Ovis-Embedding-Omni-3B | Best compared baseline | Margin |
87
  |---|---:|---:|---:|
88
  | Image | **77.55** | 73.83 | +3.72 |
89
  | Video | **64.99** | 59.37 | +5.62 |
@@ -93,17 +93,17 @@ MMEB-v3 is an omni-modal benchmark comprising **190 datasets** across image, vid
93
  | Agent | **45.52** | 39.42 | +6.10 |
94
  | **All 190 datasets** | **58.46** | 53.27 | +5.19 |
95
 
96
- Across the 31 aggregate and sub-task entries in the complete comparison, Ovis-Embedding-Omni-3B ranks first on 22 and second on 8. MultiConIR is the only entry on which it falls outside the top two.
97
 
98
  <p align="center">
99
- <img src="./ovis_blog_table1.png" alt="Complete MMEB-v3 comparison for Ovis-Embedding-Omni-3B and four omni-modal embedding baselines" width="100%"/>
100
  </p>
101
 
102
  *Scores are percentages and higher is better. Red marks the best result in each row, underlining marks the second best, and Ovis scores are bold. The overall score is the unweighted average over all 190 MMEB-v3 datasets. MMEB-v3 primarily uses Hit@1 for image, video, audio, and agent tasks and nDCG@5 for text and visual-document retrieval.*
103
 
104
  ### Additional benchmark results
105
 
106
- | Benchmark | Ovis-Embedding-Omni-3B | Evaluation scope |
107
  |---|---:|---|
108
  | MAEB (beta) | **57.29** | Mean over 30 audio embedding tasks |
109
  | MVEB (beta) | **61.77** | Mean over 23 video and audio-video embedding tasks |
@@ -145,4 +145,4 @@ This model is released under the Apache 2.0 license.
145
  - [Ovis-Omni-Embedding GitHub repository](https://github.com/ATH-MaaS/Ovis-Omni-Embedding)
146
  - [Ovis-VL-Embedding GitHub repository](https://github.com/ATH-MaaS/Ovis-VL-Embedding)
147
 
148
- This repository contains the weights for **Ovis-Embedding-Omni-3B**.
 
16
  <img alt="GitHub" src="https://img.shields.io/badge/GitHub-Ovis--Omni--Embedding-3157C8?logo=github" style="display: inline-block; vertical-align: middle;"/>
17
  </a>
18
 
19
+ # Ovis-Omni-Embedding-3B
20
 
21
+ **Ovis-Omni-Embedding-3B** is a 3B-parameter universal embedding model for text, images, visual documents, video, audio, and interleaved multimodal inputs. It maps every supported input type into one coherent representation space, enabling any-to-any retrieval with a single encoder.
22
 
23
  The model is initialized from **Qwen2.5-Omni-3B**. Rather than attaching separate modality-specific embedding towers, it retains the native text tokenizer, vision encoder, audio encoder, and shared Thinker backbone. The speech-generation Talker and language-modeling head are removed, and the final-layer hidden state at the last non-padding token is used directly as the retrieval embedding.
24
 
25
+ Ovis-Omni-Embedding-3B is designed for multimodal search, retrieval-augmented generation, recommendation, visual-document retrieval, video and audio search, and agentic retrieval over tools, interfaces, and memory.
26
 
27
  ## Model highlights
28
 
 
38
  ## Architecture
39
 
40
  <p align="center">
41
+ <img src="./ovis_embedding_model_architecture.png" alt="Architecture of Ovis-Omni-Embedding-3B and Ovis-Embedding-VL-2B/9B" width="100%"/>
42
  </p>
43
 
44
+ *Figure 2 from the technical report. Ovis-Omni-Embedding-3B encodes text, visual, and audio inputs as an interleaved token sequence processed by the Qwen2.5-Omni Thinker with TMRoPE. The final-layer hidden state at the last non-padding token becomes the retrieval embedding, with no modality-specific projection heads.*
45
 
46
  An input is formatted with a retrieval instruction through the native processor and chat template. Text, visual, and acoustic tokens are then processed as one interleaved sequence by the shared causal Transformer. Qwen2.5-Omni's time-aligned multimodal rotary position embedding preserves temporal alignment between audio and video.
47
 
 
67
 
68
  ## Retrieval interface
69
 
70
+ Ovis-Omni-Embedding-3B is a bi-encoder, not a cross-encoder:
71
 
72
  1. Pair the query with the task instruction and format it with the model's native processor and chat template.
73
  2. Encode queries and candidates independently.
 
81
 
82
  ### MMEB-v3
83
 
84
+ MMEB-v3 is an omni-modal benchmark comprising **190 datasets** across image, video, visual-document, text, audio, and agent retrieval. Ovis-Omni-Embedding-3B achieves **58.46 overall**, outperforming the strongest compared baseline by **5.19 points**, and ranks first on the aggregate score of every modality group.
85
 
86
+ | Group | Ovis-Omni-Embedding-3B | Best compared baseline | Margin |
87
  |---|---:|---:|---:|
88
  | Image | **77.55** | 73.83 | +3.72 |
89
  | Video | **64.99** | 59.37 | +5.62 |
 
93
  | Agent | **45.52** | 39.42 | +6.10 |
94
  | **All 190 datasets** | **58.46** | 53.27 | +5.19 |
95
 
96
+ Across the 31 aggregate and sub-task entries in the complete comparison, Ovis-Omni-Embedding-3B ranks first on 22 and second on 8. MultiConIR is the only entry on which it falls outside the top two.
97
 
98
  <p align="center">
99
+ <img src="./ovis_blog_table1.png" alt="Complete MMEB-v3 comparison for Ovis-Omni-Embedding-3B and four omni-modal embedding baselines" width="100%"/>
100
  </p>
101
 
102
  *Scores are percentages and higher is better. Red marks the best result in each row, underlining marks the second best, and Ovis scores are bold. The overall score is the unweighted average over all 190 MMEB-v3 datasets. MMEB-v3 primarily uses Hit@1 for image, video, audio, and agent tasks and nDCG@5 for text and visual-document retrieval.*
103
 
104
  ### Additional benchmark results
105
 
106
+ | Benchmark | Ovis-Omni-Embedding-3B | Evaluation scope |
107
  |---|---:|---|
108
  | MAEB (beta) | **57.29** | Mean over 30 audio embedding tasks |
109
  | MVEB (beta) | **61.77** | Mean over 23 video and audio-video embedding tasks |
 
145
  - [Ovis-Omni-Embedding GitHub repository](https://github.com/ATH-MaaS/Ovis-Omni-Embedding)
146
  - [Ovis-VL-Embedding GitHub repository](https://github.com/ATH-MaaS/Ovis-VL-Embedding)
147
 
148
+ This repository contains the weights for **Ovis-Omni-Embedding-3B**.