mikun0202/Chibi-VLM-3B-VQA-10K-v0.1

Chibi-VLM-3B-VQA-10K

Experimental English short-answer VQA. “3B” is a project label; component parameter counts are in parameter_counts.json. This release contains a projector and LoRA adapter, not standalone merged model weights.

Architecture

Chibi-VLM-3B-VQA-10K-v0.1 combines:

• Language model and LoRA base: openbmb/MiniCPM5-2B
• Vision encoder: google/siglip2-base-patch16-384
• Vision-language connector: custom two-layer MLP projector
• Language adaptation: LoRA applied to MiniCPM5-2B

The LoRA adapter must be loaded on top of openbmb/MiniCPM5-2B. SigLIP2 is loaded separately as the frozen vision encoder.

Training

• 10,000 distinct training images, one human QA per image; 500 validation and 500 held-out images.
• Custom image-disjoint split from lmms-lab-encoder/VQAv2 / validation. Not an official VQAv2 benchmark. Do not evaluate against the full official validation split as unseen data.
• Dataset revision: 32665d35052eb4a6d4414851c3c829a72754915a. Model revisions in config.json.
• Frozen SigLIP2, 64 pooled 2D tokens, projector-only 1 epoch, projector + LoRA 2 epochs.
• 2-GPU DDP, NF4 base weights, FP16 autocast, LoRA rank 16; effective batch 16.
• Selected checkpoint: {'stage': 2, 'epoch': 2, 'val_loss': 1.1926671148115509}.

Measured evaluation

Strict normalized exact match: 37.00%. Same 500 questions with different images: 29.20%. Paired gap: +7.80%; bootstrap CI in metrics.json. This is not official 10-annotator consensus scoring. No Japanese VQA, OCR, accessibility, or VLA performance is established.

Inference

Install requirements.txt in a CUDA PyTorch environment. Download this repo and use its loader:

from inference import ChibiVLM
model = ChibiVLM("/path/to/downloaded/repo")
print(model.answer("example.jpg", "What is on the table?"))

The loader downloads the pinned upstream models. AutoModel.from_pretrained(this_repo) alone is unsupported.

Provenance

Upstream models and dataset: MiniCPM5, SigLIP2, VQAv2 mirror. Their upstream terms and underlying image rights remain applicable. Images/QA records are not included in this release.

Support

Support for the author's work is optional and does not affect model access.