upload figures and README

.gitattributes

@@ -34,3 +34,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+figures/global_pool.png filter=lfs diff=lfs merge=lfs -text
+figures/local_pool.png filter=lfs diff=lfs merge=lfs -text
+figures/zero_shot_image.png filter=lfs diff=lfs merge=lfs -text
+figures/zero_shot_video.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,113 @@
----
-license: apache-2.0
----
+---
+base_model:
+- Qwen/Qwen2-VL-7B-Instruct
+datasets:
+- TIGER-Lab/M-BEIR
+language:
+- en
+license: apache-2.0
+pipeline_tag: any-to-any
+library_name: transformers
+---
+
+## U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs
+
+This repository contains the official model checkpoints and inference code for **U-MARVEL**, presented in the paper [U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs](https://huggingface.co/papers/2507.14902).
+
+Universal multimodal retrieval (UMR) addresses complex retrieval tasks involving diverse modalities for both queries and candidates. Despite the success of state-of-the-art methods based on multimodal large language models (MLLMs) using contrastive learning principles, the mechanisms underlying their retrieval capabilities remain largely unexplored. This gap potentially leads to suboptimal performance and limited generalization ability.
+
+In this study, we systematically analyze the key factors driving effective embedding learning for UMR using MLLMs. We implement a general MLLM-based embedding learning pipeline and investigate contributors to high-performing universal retrieval systems. Our analysis covers various aspects of embedding generation and training strategies, including progressive transition, hard negative mining, and re-ranker distillation. Our findings reveal that often-overlooked factors can significantly impact model performance.
+
+Building on these insights, we introduce U-MARVEL (Universal Multimodal Retrieval via Embedding Learning), a unified framework that outperforms state-of-the-art competitors on the M-BEIR benchmark in supervised settings and demonstrates strong zero-shot performance on tasks such as composed image retrieval and text-to-video retrieval. These results highlight the generalization potential of our framework across various embedding-based retrieval tasks, providing valuable insights for future research.
+
+## Model Checkpoints
+
+```
+├── checkpoints
+│   ├── hf_models
+│   │   └── Qwen2-VL-7B-Instruct
+│   │   └── Qwen3-VL-4B-Instruct
+│   └── U-MARVEL-Qwen2VL-7B-Instruct
+│   └── U-MARVEL-Qwen3VL-4B-Instruct
+```
+
+- [U-MARVEL-Qwen2VL-7B-Instruct](https://huggingface.co/TencentBAC/U-MARVEL-Qwen2VL-7B-Instruct) 🤗
+- [U-MARVEL-Qwen3VL-4B-Instruct](https://huggingface.co/TencentBAC/U-MARVEL-Qwen3VL-4B-Instruct) 🤗
+- Inference code available at: [U-MARVEL-inference](https://github.com/chaxjli/U-MARVEL)
+## Requirements
+
+To install requirements:
+
+```setup
+pip install -r requirements.txt
+```
+
+### Data Preparation
+
+Download Qwen2-VL-7B and place it in `./checkpoints/hf_models/Qwen2-VL-7B-Instruct`
+
+Download Qwen3-VL-4B and place it in `./checkpoints/hf_models/Qwen3-VL-4B-Instruct`
+
+For NLI dataset, please refer to [link](https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse)
+
+For multimodal instruction tuning datset, please refer to [M-BEIR](https://huggingface.co/datasets/TIGER-Lab/M-BEIR)
+
+After downloading all of them, organize the data as follows in `./data`
+
+```
+├── data    
+│    ├── M-BEIR
+│    ├── nli_for_simcse.csv
+│    ├── rerank_data_for_training
+│    ├── flickr
+│    ├── coco
+│    ├── sharegpt4v
+│    ├── Urban1K
+│    ├── circo
+│    ├── genecis
+│    ├── vist
+│    ├── visdial
+│    ├── ccneg
+│    ├── sugar-crepe
+│    ├── MSVD
+│    └── msrvtt
+```
+
+## Evaluation
+
+To evaluate our model on M-BEIR, run:
+
+```bash
+python scripts/vtools_eval_mbeir_model.py  # Evaluate locally  
+sh scripts/eval_mbeir_global.sh            # Evaluate globally  
+sh scripts/eval_zeroshot.sh                # Evaluate zero-shot
+```
+
+## Model Performance
+
+The proposed U-MARVEL framework establishes new state-of-the-art performance across both
+single-model architectures and recall-then-rerank approaches on M-BEIR benchmark.
+
+<img src="./figures/local_pool.png" alt="M-BEIR-Local" width="700" height="auto">
+
+<img src="./figures/global_pool.png" alt="M-BEIR-Global" width="700" height="auto">
+
+<img src="./figures/zero_shot_image.png" alt="M-BEIR-Zero-shot_image" width="700" height="auto">
+
+<img src="./figures/zero_shot_video.png" alt="M-BEIR-Zero-shot_video" width="700" height="auto">
+
+
+## Acknowledgements
+
+Many thanks to the code bases from **[LamRA](https://github.com/Code-kunkun/LamRA)** .
+
+## Citation
+If you use this code for your research or project, please cite:
+```latex
+@article{li2025umarvel,
+  title={U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs},
+  author={Li, Xiaojie and Li, Chu and Chen, Shi-Zhe and Chen, Xi},
+  journal={arXiv preprint arXiv:2507.14902},
+  year={2025}
+}
+```