---
language:
- id
- en
license: gemma
library_name: transformers
tags:
- t5gemma-2
- unsloth
- vision
- multimodal
- siglip
- orpo
- indonesian
- custom-seq2seq
pipeline_tag: image-text-to-text
base_model: daruokta/t5gemma-2-4b-4b-instruct-chat-indo-v4-vision-cangkok
---

# T5Gemma-2-4B-4B Vision Enhanced (Indonesian Multimodal Instruction-Tuned Model)

**T5Gemma-2-4B-4B Vision Enhanced** adalah model multimodal (Vision-Language / Image-to-Text) berarsitektur *Encoder-Decoder* (Seq2Seq) yang dirancang khusus untuk memahami dan merespons instruksi berbasis **gambar dan teks dalam Bahasa Indonesia**.

Model ini dikembangkan dengan memisahkan dan memverifikasi komponen *Vision Tower* SigLIP 400M yang dicangkokkan ke *backbone* **T5Gemma-2-4B**, kemudian dilatih secara bertahap menggunakan metode **SFT (Supervised Fine-Tuning)** dan **ORPO (Odds Ratio Preference Optimization)** melalui kerangka kerja **Unsloth**.

---

## 🌟 Highlight Utama & Solusi Bebas Catastrophic Forgetting

1. 🛡️ **Pencegahan Catastrophic Forgetting (Interleaved Text Retention):**
   - Mengintegrasikan 100 percakapan utuh retensi teks murni (*multiturn intact*) dari repo [`daruokta/t5gemma2-indonesia-chat-formatted`](https://huggingface.co/datasets/daruokta/t5gemma2-indonesia-chat-formatted) (`chat_sft` & `indoqa_sft`).
   - Berhasil memecahkan masalah degradasi teks murni (Exact Match melonjak dari **0.00% pada eksperimen lama ➔ 33.33% pada eksperimen Enhanced ini**).

2. 🎛️ **Split Learning Rates Terpisah (Split LR):**
   - **Decoder (LoRA)**: `1e-6` (`LEARNING_RATE * 0.2`) untuk menjaga representasi bahasa yang matang.
   - **Multi-Modal Projector (Full-FT)**: `2.5e-7` (`LEARNING_RATE * 0.05`) untuk jembatan pemetaan visual yang halus.
   - **Vision Tower (SigLIP)**: `0.0` (Frozen).

3. 🔒 **Stabilitas Gradiensi Ketat (`max_grad_norm = 5.0` & `WEIGHT_DECAY = 0.1`):**
   - Menahan lonjakan grad norm ekstrem (sebelumnya sempat meledak ke 121.08) sehingga pelatihan berlangsung sangat stabil.

4. 🎯 **Penyesuaian Preferensi Multimodal dengan ORPO:**
   - Dilatih pada dataset preferensi bergambar untuk mempertajam kemampuan visual (*chosen vs rejected*) tanpa mengorbankan kualitas bahasa Indonesia.

---

## 📊 Hasil Evaluasi & Dashboard Grafik Komparasi Lengkap

![Full Comprehensive Metrics Dashboard](comprehensive_metrics_dashboard.png)

### Perbandingan SFT & ORPO (Old vs Enhanced)

| Kategori Tahap | Metrik Evaluasi | Eksperimen Lama (`v4-vision`) | **Eksperimen Baru (`v4-vision-enhanced`)** | Status Perbaikan |
| :--- | :--- | :---: | :---: | :--- |
| **SFT Teks** | **Text-Only Validation Loss** | `3.3538` 🔴 *(Exploded)* | **`2.4578`** 🟢 | ✅ **Loss Teks Turun -26.7%** |
| **SFT Teks** | **Text-Only Perplexity** | `28.61` 🔴 *(Membengkak)* | **`11.68`** 🟢 | ✅ **Perplexity Membaik -59.2%** |
| **SFT Teks** | **Text-Only ROUGE-1** | `27.20%` 🔴 *(Drop)* | **`60.09%`** 🟢 | ✅ **ROUGE-1 Naik +120.9%** |
| **SFT Teks** | **Text-Only Exact Match** | **`0.00%`** 🔴 *(Lupa)* | 🔥 **`33.33%`** 🟢 | 🎉 **Forgetting 100% Teratasi** |
| **SFT Vision** | **Multimodal Loss (Gambar)** | `2.9072` | **`2.8800`** 🟢 | ✅ **Loss Visual Terjaga & Menurun** |
| **SFT Vision** | **Multimodal ROUGE-1** | `37.65%` | **`42.83%`** 🟢 | ✅ **Skor Visual Naik ke Puncak** |
| **ORPO Teks** | **Text-Only Validation Loss**| - | 🔥 **`0.8591`** 🟢 | 🎉 **Loss Teks Terendah Rekor Baru** |
| **ORPO Teks** | **Text-Only Perplexity** | - | 🔥 **`2.36`** 🟢 | 🎉 **PPL Terendah Rekor Baru** |
| **ORPO Teks** | **Text-Only ROUGE-1** | - | 🔥 **`61.65%`** 🟢 | 🎉 **ROUGE-1 Puncak** |
| **ORPO Vision**| **Multimodal Loss (Gambar)** | `1.5314` | **`1.6151`** 🟢 | ✅ **Preference Alignment Presisi** |

---

## 📚 Dataset Yang Digunakan

1. **Multimodal Vision Dataset (SFT & ORPO):**
   - Repo: [`daruokta/t5gemma2-indonesia-vision-formatted`](https://huggingface.co/datasets/daruokta/t5gemma2-indonesia-vision-formatted)
   - Berisi dataset pasangan Gambar + Teks instruksi Bahasa Indonesia (SFT) dan sampel preferensi multimodal (*chosen vs rejected*).

2. **Text Retention Dataset (Interleaved SFT):**
   - Repo: [`daruokta/t5gemma2-indonesia-chat-formatted`](https://huggingface.co/datasets/daruokta/t5gemma2-indonesia-chat-formatted)
   - Config: `chat_sft` (100 percakapan utuh *multiturn*) & `indoqa_sft` (100 sampel *single-turn*).

---

## 📋 Cara Penggunaan (Inference Multimodal)

```python
import torch
from PIL import Image
from transformers import AutoProcessor
from unsloth import FastVisionModel

model_id = "daruokta/t5gemma-2-4b-4b-instruct-chat-indo-v4-vision-enhanced"

# Load model dan processor
model, tokenizer = FastVisionModel.from_pretrained(
    model_name=model_id,
    load_in_4bit=True,
    device_map="auto"
)
FastVisionModel.for_inference(model)

processor = AutoProcessor.from_pretrained(model_id)

# Buka gambar contoh
image = Image.open("contoh_gambar.jpg").convert("RGB")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "Jelaskan isi gambar ini secara rinci dalam Bahasa Indonesia."}
        ]
    }
]

prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=image, return_tensors="pt").to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    use_cache=True,
    temperature=0.7,
    min_p=0.1
)

response = processor.batch_decode(outputs, skip_special_tokens=True)
print(response[0])
```

---

## 🔧 Spesifikasi & Hyperparameter Pelatihan

| Parameter | Nilai Fase Vision SFT | Nilai Fase Vision ORPO |
| :--- | :--- | :--- |
| **Base Model** | `daruokta/t5gemma-2-4b-4b-instruct-chat-indo-v4-vision-cangkok` | `SFT Final Adapter Checkpoint` |
| **Vision Encoder** | SigLIP 400M (Frozen) | SigLIP 400M (Frozen) |
| **Multi-Modal Projector** | Full Fine-Tuning (`2.5e-7` LR) | Full Fine-Tuning (`2.5e-7` LR) |
| **Decoder LoRA Rank (r)** | `256` | `256` |
| **Decoder LoRA Alpha (α)**| `512` | `512` |
| **Decoder Learning Rate** | `1e-6` | `1e-6` |
| **Epochs** | `2` | `1` |
| **Weight Decay** | `0.1` | `0.1` |
| **Max Grad Norm** | `5.0` | `5.0` |
| **Effective Batch Size** | `2` per device × `32` grad accumulation (`64`) | `2` per device × `32` grad accumulation (`64`) |
| **Max Context Length** | Source: `16384` / Target: `2048` | Source: `16384` / Target: `2048` |

---

## 🏗️ Arsitektur Multimodal T5Gemma-2 Vision

![T5Gemma-2 Vision Architecture](architecture_diagram.png)

---

## 📚 Referensi & Publikasi Ilmiah

1. **T5Gemma 2: Seeing, Reading, and Understanding Longer** (Google DeepMind, 2025) — [arXiv:2512.14856](https://arxiv.org/abs/2512.14856)
2. **ORPO: Easy Harmless Alignment with Odds Ratio Preference Optimization** (2024) — [arXiv:2403.07691](https://arxiv.org/abs/2403.07691)
3. **SigLIP: Sigmoid Loss for Language Image Pre-training** (Zhai et al., 2023) — [arXiv:2303.15343](https://arxiv.org/abs/2303.15343)
4. **Gemma 3 Technical Report** (Google DeepMind, 2025) — [arXiv:2503.19786](https://arxiv.org/abs/2503.19786)