Qwen3-VL-2B-Instruct OpenVINO INT4

This repository contains a local OpenVINO IR export of Qwen/Qwen3-VL-2B-Instruct converted for OpenVINO GenAI inference and compressed to INT4 weights.

Requirements

This model is intended for OpenVINO GenAI VLM inference.

pip install -U openvino openvino-tokenizers openvino-genai pillow numpy

Download

hf download shawnxhong/Qwen3-VL-2B-Instruct-ov-int4 --local-dir Qwen3-VL-2B-Instruct-ov-int4

Included files

OpenVINO GenAI tokenizer and detokenizer
OpenVINO language model
OpenVINO text embeddings model
OpenVINO vision embeddings model
OpenVINO vision merger and positional embedding models

Minimal local inference

from pathlib import Path

import numpy as np
import openvino as ov
import openvino_genai
from PIL import Image

model_dir = Path("Qwen3-VL-2B-Instruct-ov-int4")
image_path = Path("example.png")

pipe = openvino_genai.VLMPipeline(str(model_dir), "CPU")

image = Image.open(image_path).convert("RGB")
image_tensor = ov.Tensor(np.array(image)[None])

generation_config = pipe.get_generation_config()
generation_config.max_new_tokens = 128
generation_config.do_sample = False

result = pipe.generate(
    "Read the image and return the visible text.",
    image_tensor,
    generation_config,
)

print(result.texts[0])

Minimal script

import openvino as ov
import openvino_genai
from PIL import Image
import numpy as np

pipe = openvino_genai.VLMPipeline("Qwen3-VL-2B-Instruct-ov-int4", "CPU")
image = ov.Tensor(np.array(Image.open("example.png").convert("RGB"))[None])
result = pipe.generate("Read the image and return the visible text.", image, max_new_tokens=128)
print(result.texts[0])

Benchmark with OpenVINO GenAI `llm_bench`

If you have the openvino.genai source tree locally:

python tools/llm_bench/benchmark.py \
  -m Qwen3-VL-2B-Instruct-ov-int4 \
  -d CPU \
  -t visual_text_gen \
  -i example.png \
  -p "Read the image and return the visible text." \
  -n 3 \
  -ic 128 \
  -r benchmark_report.csv \
  -rj benchmark_report.json

Notes

This export targets OpenVINO GenAI VLMPipeline.
The model was converted from Qwen/Qwen3-VL-2B-Instruct to OpenVINO IR and compressed to INT4.
For OCR-style usage, short prompts like Read the image and return the visible text. work well as a baseline.

Downloads last month: 19