Instructions to use TencentBAC/U-MARVEL-Qwen2VL-7B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use TencentBAC/U-MARVEL-Qwen2VL-7B-Instruct with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForSeq2SeqLM processor = AutoProcessor.from_pretrained("TencentBAC/U-MARVEL-Qwen2VL-7B-Instruct") model = AutoModelForSeq2SeqLM.from_pretrained("TencentBAC/U-MARVEL-Qwen2VL-7B-Instruct", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -33,54 +33,16 @@ Building on these insights, we introduce U-MARVEL (Universal Multimodal Retrieva
|
|
| 33 |
|
| 34 |
- [U-MARVEL-Qwen2VL-7B-Instruct](https://huggingface.co/TencentBAC/U-MARVEL-Qwen2VL-7B-Instruct) π€
|
| 35 |
- [U-MARVEL-Qwen3VL-4B-Instruct](https://huggingface.co/TencentBAC/U-MARVEL-Qwen3VL-4B-Instruct) π€
|
| 36 |
-
-
|
| 37 |
-
##
|
| 38 |
|
| 39 |
-
To install
|
| 40 |
-
|
| 41 |
-
```setup
|
| 42 |
-
pip install -r requirements.txt
|
| 43 |
-
```
|
| 44 |
-
|
| 45 |
-
### Data Preparation
|
| 46 |
-
|
| 47 |
-
Download Qwen2-VL-7B and place it in `./checkpoints/hf_models/Qwen2-VL-7B-Instruct`
|
| 48 |
-
|
| 49 |
-
Download Qwen3-VL-4B and place it in `./checkpoints/hf_models/Qwen3-VL-4B-Instruct`
|
| 50 |
-
|
| 51 |
-
For NLI dataset, please refer to [link](https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse)
|
| 52 |
-
|
| 53 |
-
For multimodal instruction tuning datset, please refer to [M-BEIR](https://huggingface.co/datasets/TIGER-Lab/M-BEIR)
|
| 54 |
-
|
| 55 |
-
After downloading all of them, organize the data as follows in `./data`
|
| 56 |
-
|
| 57 |
-
```
|
| 58 |
-
βββ data
|
| 59 |
-
β βββ M-BEIR
|
| 60 |
-
β βββ nli_for_simcse.csv
|
| 61 |
-
β βββ rerank_data_for_training
|
| 62 |
-
β βββ flickr
|
| 63 |
-
β βββ coco
|
| 64 |
-
β βββ sharegpt4v
|
| 65 |
-
β βββ Urban1K
|
| 66 |
-
β βββ circo
|
| 67 |
-
β βββ genecis
|
| 68 |
-
β βββ vist
|
| 69 |
-
β βββ visdial
|
| 70 |
-
β βββ ccneg
|
| 71 |
-
β βββ sugar-crepe
|
| 72 |
-
β βββ MSVD
|
| 73 |
-
β βββ msrvtt
|
| 74 |
-
```
|
| 75 |
-
|
| 76 |
-
## Evaluation
|
| 77 |
-
|
| 78 |
-
To evaluate our model on M-BEIR, run:
|
| 79 |
|
| 80 |
```bash
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
|
|
|
| 84 |
```
|
| 85 |
|
| 86 |
## Model Performance
|
|
@@ -104,10 +66,10 @@ Many thanks to the code bases from **[LamRA](https://github.com/Code-kunkun/LamR
|
|
| 104 |
## Citation
|
| 105 |
If you use this code for your research or project, please cite:
|
| 106 |
```latex
|
| 107 |
-
@
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
}
|
| 113 |
```
|
|
|
|
| 33 |
|
| 34 |
- [U-MARVEL-Qwen2VL-7B-Instruct](https://huggingface.co/TencentBAC/U-MARVEL-Qwen2VL-7B-Instruct) π€
|
| 35 |
- [U-MARVEL-Qwen3VL-4B-Instruct](https://huggingface.co/TencentBAC/U-MARVEL-Qwen3VL-4B-Instruct) π€
|
| 36 |
+
- Code available at: [U-MARVEL](https://github.com/chaxjli/U-MARVEL)
|
| 37 |
+
## π Demo
|
| 38 |
|
| 39 |
+
To get started, first create a virtual environment and install the required dependencies:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
```bash
|
| 42 |
+
conda create -n u-marvel python=3.9 -y
|
| 43 |
+
conda activate u-marvel
|
| 44 |
+
pip install -r requirements_qwen2_vl.txt
|
| 45 |
+
python demo.py
|
| 46 |
```
|
| 47 |
|
| 48 |
## Model Performance
|
|
|
|
| 66 |
## Citation
|
| 67 |
If you use this code for your research or project, please cite:
|
| 68 |
```latex
|
| 69 |
+
@inproceedings{li2026umarvel,
|
| 70 |
+
title={U-{MARVEL}: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with {MLLM}s},
|
| 71 |
+
author={Xiaojie Li and Chu Li and Shi-Zhe Chen and Xi Chen},
|
| 72 |
+
booktitle={The Fourteenth International Conference on Learning Representations},
|
| 73 |
+
year={2026}
|
| 74 |
}
|
| 75 |
```
|