---
datasets:
- HuggingFaceM4/WebSight
language:
- en
base_model:
- Qwen/Qwen2.5-VL-3B-Instruct
library_name: transformers
tags:
- code
- multimodal
- ui
- webdesign
- merge
pipeline_tag: image-text-to-text
---

# Qwen2.5-VL-3B-Instruct-Img2Code 🎨→💻

![image](qwen2_5.png)

**Qwen2.5-VL-3B-Instruct-Img2Code** is a fine-tuned version of [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) specialized in **generating clean, functional HTML/CSS code from webpage screenshots**.

## 🎯 Model Description

This model transforms webpage screenshots into semantically correct HTML/CSS code, supporting:
- **Tailwind CSS** integration
- **Responsive design** layouts
- **Semantic HTML5** structure
- Complex UI components (navigation bars, forms, cards, grids)
- Modern CSS features (flexbox, grid, animations)

## 🔧 Training Details

- **Base model:** `Qwen/Qwen2.5-VL-3B-Instruct`  
- **Method:** LoRA (r=8, alpha=16)
- **Target modules:** `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`
- **Dataset:** [HuggingFaceM4/WebSight](https://huggingface.co/datasets/HuggingFaceM4/WebSight)
  - 8,100 training examples
  - 900 evaluation examples
- **Configuration:**
  - Total steps: 438
  - Learning rate: 2e-4 (cosine with warmup)
  - Gradient accumulation: 16
  - Image resolution: 256×28×28 to 512×28×28 pixels
- **Hardware:** NVIDIA L4 24GB VRAM
- **Training time:** ~5.5 hours

## 📊 Performance Metrics

| Metric | Value |
|--------|-------|
| **Final Eval Loss** | 0.180 |
| **Final Eval Accuracy** | 94.6% |
| **Best Train Loss** | 0.133 |
| **Best Train Accuracy** | 95.9% |

## 💻 Usage

### Installation

```bash
pip install transformers torch pillow qwen-vl-utils torchvision
# Optional but recommended:
pip install flash-attn --no-build-isolation
```

### Basic Inference

```python
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
from PIL import Image
import torch

# Load model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # Requires flash-attn
    device_map="auto"
)

# Without flash-attn:
# model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
#     "Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code",
#     dtype="auto",
#     device_map="auto"
# )

# Load processor
processor = AutoProcessor.from_pretrained(
    "Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code",
    min_pixels=256*28*28,
    max_pixels=1280*28*28
)

# Load image
image = Image.open("screenshot.jpg")

# Prepare messages
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image,
            },
            {"type": "text", "text": "Generate the HTML/CSS code for this webpage screenshot."},
        ],
    }
]


text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)
```

### Streaming Inference

```python
from transformers import TextIteratorStreamer
from threading import Thread

# ... (previous loading code)

streamer = TextIteratorStreamer(
    processor,
    skip_prompt=True,
    skip_special_tokens=True
)

generation_kwargs = dict(
    inputs,
    streamer=streamer,
    max_new_tokens=2048
)

thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()

print("Generating code:")
for new_text in streamer:
    print(new_text, end="", flush=True)

thread.join()
```

### Production Deployment with vLLM

**Start server**

```bash
vllm serve Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key dummyapikey \
  --mm-encoder-tp-mode data \
  --limit-mm-per-prompt '{"image":2,"video":0}' \
  --max-model-len=16384 \
  --gpu-memory-utilization=0.90
```


**Request to the server from the OpenAI client**

```python
from openai import OpenAI
import base64

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

client = OpenAI(api_key="dummyapikey", base_url="http://127.0.0.1:8000/v1")
image_base64 = encode_image("screenshot.jpg")

stream = client.chat.completions.create(
    model="Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Generate the HTML/CSS code for this webpage screenshot."},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
        ]
    }],
    max_tokens=2048,
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

**vLLM Benefits:** 20-30x faster, OpenAI-compatible API, continuous batching.

## 🚀 Capabilities & Limitations

### ✅ Supported Features
- Landing pages, navigation bars, card layouts, forms
- Dashboards with data visualization placeholders
- Multi-column responsive layouts
- Tailwind CSS and modern CSS (gradients, shadows, animations)
- Semantic HTML5 structure

### ⚠️ Limitations
- **Static HTML/CSS only** (no JavaScript logic)
- Uses placeholder images (doesn't extract actual images)
- OCR limitations may affect text accuracy
- Requires manual review before production use
- **Best for:** Starting point for development, not production-ready code

## 🔗 Related Products

### **Aquiles-RAG** - High-Performance Retrieval-Augmented Generation
- **Repository:** https://github.com/Aquiles-ai/Aquiles-RAG
- **PyPI:** `pip install aquiles-rag`
- **Features:**
  - Vector search (Redis HNSW, Qdrant, PostgreSQL pgvector)
  - FastAPI REST API
  - Embedding-agnostic architecture
  - Sync & async Python clients
  - Interactive setup wizard
  - Optional re-ranking

**Perfect for:** Building intelligent code search systems, documentation assistants, or UI component libraries with semantic search capabilities.

## 📚 Dataset

**WebSight** (HuggingFace M4): High-quality webpage screenshots paired with source code, including landing pages, dashboards, blogs, and e-commerce sites.

## 📄 Citation

```bibtex
@misc{aquiles-qwen-img2code,
  author = {Aquiles-ai},
  title = {Qwen2.5-VL-3B-Instruct-Img2Code: Automated Webpage Screenshot to Code Generation},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Aquiles-ai/Qwen2.5-VL-3B-Instruct-Img2Code}
}
```

## 🙏 Acknowledgments

- **Qwen Team** for the excellent base model [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct)
- **HuggingFace M4** for the [WebSight dataset](https://huggingface.co/datasets/HuggingFaceM4/WebSight)

## 📜 License

Same license as the base model Qwen2.5-VL-3B-Instruct

**Contact:** https://aquiles-ai.vercel.app
**Version:** 1.0  
**Last Updated:** October 2025