---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- pytorch
- safetensors
- vision-language
- visual-question-answering
pipeline_tag: visual-question-answering
base_model:
- Qwen/Qwen3-0.6B-Base
model-index:
- name: MicroLLaVA-Qwen3-0.6B-base-siglip2-so400m
  results:
  - task:
      type: visual-question-answering
      name: VQAv2
    dataset:
      name: VQAv2
      type: vqav2
    metrics:
    - name: Overall Accuracy
      type: accuracy
      value: 78.52
    - name: Yes/No Accuracy
      type: accuracy
      value: 91.56
    - name: Number Accuracy
      type: accuracy
      value: 65.69
    - name: Other Accuracy
      type: accuracy
      value: 70.28
    source:
      name: Internal Evaluation on VQAv2 test-dev
      url: https://visualqa.org/download.html
---
# 🦙📷 MicroLLaVA-Qwen3-0.6B-base-siglip2-so400m

**A compact yet competitive open-source vision-language model trained from scratch on a single RTX 4090.**

This is a ~1B parameter model that performs **on par with the original [LLaVA-1.5-7B](https://huggingface.co/liuhaotian/llava-v1.5-7b)**,
making it a serious candidate for **visual question answering tasks**—both for **everyday users** and **researchers focused on efficient multimodal architectures**.

It is also **well suited for Edge AI applications**, such as **on-device visual question answering**, thanks to its small size and fast inference performance.

---

## 📌 Model Summary

**`keeeeenw/MicroLLaVA-Qwen3-0.6B-base-siglip2-so400m`** combines the strengths of:

- **[Qwen3-0.6B-base](https://huggingface.co/Qwen/Qwen3-0.6B)** – a powerful open LLM from Alibaba
- **[siglip2-so400m-patch14-384](https://huggingface.co/your/siglip2-model-link)** – a strong visual encoder with patch size 14 and 384 resolution

Trained using a [modified TinyLLaVA Factory fork](https://github.com/keeeeenw/TinyLLaVA_Factory/tree/cu12_qwen3), this model reaches **78.52% overall on VQAv2 test-dev**, outperforming:

- 🥈 `Zhang199/TinyLLaVA-Qwen2-0.5B-SigLIP`: 72.33%  
- 🥉 On par with `LLaVA-1.5-7B`: ~78.5%

See https://huggingface.co/Zhang199/TinyLLaVA-Qwen2-0.5B-SigLIP#result for evaluation of the original models from TinyLLaVa and LLaVa authors.

---

## 🖼️ Example Inference

**Code**:

```python

# Tested with transformers: 4.55.2, torch: 2.8.0, and torchvision: 0.23.0
# Older versions of the transformers do not support qwen3 model.
# pip install transformers torch torchvision

from transformers import AutoTokenizer, AutoModelForCausalLM

hf_path = 'keeeeenw/MicroLlava-Qwen3-0.6B-base-siglip2-so400m'
model = AutoModelForCausalLM.from_pretrained(hf_path, trust_remote_code=True)
# model.cuda() # if no cuda is used, it takes ~78 seconds inference on my AMD 5950 CPU. 
config = model.config
tokenizer = AutoTokenizer.from_pretrained(hf_path, use_fast=False, model_max_length = config.tokenizer_model_max_length,padding_side = config.tokenizer_padding_side)

prompt="What are the things I should be cautious about when I visit here?"
image_url="https://llava-vl.github.io/static/images/view.jpg"
output_text, genertaion_time = model.chat(prompt=prompt,
                                          image=image_url,
                                          tokenizer=tokenizer)

print('model output:', output_text)
print('runing time:', genertaion_time)

```

**Prompt**:  
🧠 *"What are the things I should be cautious about when I visit here?"*

**Image**:  
![example image](https://llava-vl.github.io/static/images/view.jpg)

**Model Output**:
When visiting this wooden pier on the lake, there are a few things to be cautious about. First, be aware of the water depth, as the water may be shallow or choppy, which could pose a risk of falling into the water or getting caught in a strong current. Second, be mindful of the weather conditions, as the image shows a cloudy day, which might indicate the possibility of rain or strong winds. These conditions can make the experience more challenging and potentially dangerous. Lastly, be cautious about the presence of any wildlife in the area, as the lake and its surroundings may attract various animals. Always follow safety guidelines and be respectful of the natural environment when visiting this picturesque location.

---

## ⚙️ Training Details

- 🧠 **Total parameters**: ~1B
- 🖥️ **Hardware**: Single NVIDIA RTX 4090 (24GB VRAM)
- ⏱️ **Total training time**: ~24 hours
  - Stage 1 (pretraining): ~8 hours
  - Stage 2 (fine-tuning): ~12 hours
- 🧾 **Training method**: Follows official [TinyLLaVA instructions](https://github.com/keeeeenw/TinyLLaVA_Factory/tree/cu12_qwen3?tab=readme-ov-file#2-train)
  - ⚠️ `ocr_vqa` was **omitted in fine-tuning** due to dataset access issues (to be re-added in next training run)

---

## ✨ Key Contributions

1. ✅ Validated **Qwen3-0.6B + SigLIP2** as an efficient, high-performance combination for vision-language tasks
2. 🛠️ Upgraded TinyLLaVA to support **Qwen3 models** and **latest PyTorch/transformers**
3. 🧩 Created a new **Qwen3 chat template**  
   → [`qwen3_base_template.py`](https://github.com/keeeeenw/TinyLLaVA_Factory/blob/cu12_qwen3/tinyllava/data/template/qwen3_base_template.py)
4. ⚙️ Performed hyperparameter tuning for optimal Qwen3 + SigLIP2 performance
5. 🚀 Released **standalone Hugging Face inference** support—no need to install TinyLLaVA

---

## 📊 VQAv2 Evaluation (test-dev)

| Question Type | Accuracy |
|---------------|----------|
| Yes/No        | 91.56%   |
| Number        | 65.69%   |
| Other         | 70.28%   |
| **Overall**   | **78.52%** |

> Note: Evaluation was performed on the **VQAv2 test-dev** set. It is unclear whether prior models used test-dev or test set. This model is currently being evaluated on the full test set and other benchmarks.

---

## 🔜 Upcoming Work

This model is currently undergoing evaluation on:

- VQAv2 **test** set
- GQA
- SQA
- TextVQA
- MM-VET
- POPE
- MME
- MMMU

Stay tuned for updates!

---

## 🧾 Citation

If you find this model helpful, please consider citing or referencing this repo:
```
@misc{wang2024microllama,
  title        = {MicroLLaVA: a TinyLLaVA based VLM with MicroLlama 300M for single GPU training},
  author       = {Zixiao Ken Wang},
  year         = {2025},
  url          = {https://huggingface.co/keeeeenw/MicroLlava,https://huggingface.co/keeeeenw/MicroLlava-Qwen3-0.6B-base-siglip2-so400m}
}
```

Please also support my release of https://huggingface.co/keeeeenw/MicroLlava which is based on own https://huggingface.co/keeeeenw/MicroLlama for the language capabilities.