README.md

---
license: apache-2.0
base_model:
  - Qwen/Qwen2-VL-7B-Instruct
pipeline_tag: image-text-to-text
tags:
  - image-captioning
  - reinforcement-learning
  - GRPO
---

# CIM-Qwen2-VL-7B-SFT

This model is fine-tuned from [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) using **SFT + GRPO** with the **Cross-modal Identity Mapping (CIM)** reward, as described in our CVPR 2026 paper.

[![arXiv](https://img.shields.io/badge/cs.CV-2603.01696-b31b1b?logo=arxiv&logoColor=red)](https://arxiv.org/abs/2603.01696)
[![GitHub](https://img.shields.io/badge/GitHub-CIM-blue?logo=github)](https://github.com/Jia-hn/CIM)

## Overview

CIM is a reinforcement learning framework that improves image captioning by minimizing information loss during modality conversion. It uses two reward signals — **Gallery Representation Consistency (GRC)** and **Query-gallery Image Relevance (QIR)** — to encourage LVLMs to generate fine-grained and precise captions without extra annotations.

## Usage

```python
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2VLForConditionalGeneration.from_pretrained("kkk5/CIM-Qwen2-VL-7B-SFT", torch_dtype="auto", device_map="auto")
processor = AutoProcessor.from_pretrained("kkk5/CIM-Qwen2-VL-7B-SFT")

messages = [{"role": "user", "content": [
    {"type": "image", "image": "your_image.jpg"},
    {"type": "text", "text": "Caption this image as accurately as possible, without speculation. Describe what you see."},
]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt").to(model.device)

output_ids = model.generate(**inputs, max_new_tokens=1024)
output_text = processor.batch_decode(output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0]
print(output_text)
```

## Citation

```bibtex
@inproceedings{jia2026cross,
    title     = {Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning},
    author    = {Jia, Haonan and Dong, Shichao and Dong, Xin and Sun, Zenghui and Wang, Jin and Lan, Jinsong and Zhu, Xiaoyong and Zheng, Bo and Zhang, Kaifu},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    pages     = {766--777},
    year      = {2026}
}
```