---
base_model: HuggingFaceTB/SmolVLM-256M-Instruct
datasets:
  - w4ashabii/nepali_crop_data
language:
  - en
library_name: peft
license: apache-2.0
tags:
  - vision-language
  - vlm
  - lora
  - unsloth
  - agriculture
  - crop-disease
  - smolvlm
  - image-to-text
pipeline_tag: image-text-to-text
---

# SmolVLM-256M — Nepali Crop Disease

A LoRA fine-tune of [`HuggingFaceTB/SmolVLM-256M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct) that answers **"What crop is shown in this image, and does it have any disease?"** for photos of Nepali crops. Trained on [`w4ashabii/nepali_crop_data`](https://huggingface.co/datasets/w4ashabii/nepali_crop_data), a ShareGPT-format visual-QA dataset covering 9 crops.

## Model Details

| | |
|---|---|
| **Base model** | [HuggingFaceTB/SmolVLM-256M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct) (~256M params) |
| **Fine-tuning method** | LoRA (adapters only, base weights frozen) |
| **Trainable params** | 3,769,344 / 260,254,272 (**1.45%**) |
| **Framework** | 🤗 `transformers` `Trainer` + [Unsloth](https://github.com/unslothai/unsloth) `FastVisionModel` |
| **LoRA targets** | Attention + MLP modules across both vision encoder and language model |
| **LoRA config** | r=8, alpha=16, dropout=0.05, bias="none" |
| **Precision** | fp16 |
| **Hardware** | 1x NVIDIA T4 (Google Colab) |
| **Language(s)** | English (Q&A text); images of Nepali-grown crops |
| **License** | Apache 2.0 (inherited from base model) |

## Training Data

- **Dataset:** [`w4ashabii/nepali_crop_data`](https://huggingface.co/datasets/w4ashabii/nepali_crop_data)
- **Format:** ShareGPT-style conversations (`sharegpt.jsonl`), one user turn (image + question) → one assistant turn (crop + disease answer)
- **Train examples used:** 39,987
- **Eval examples used:** 500 (held-out validation split)
- **Image preprocessing:** resized to `longest_edge=512`, image splitting disabled (not needed for single-leaf/crop photos)

## Training Procedure

| Hyperparameter | Value |
|---|---|
| Epochs | 2 |
| Per-device batch size | 8 |
| Gradient accumulation steps | 2 |
| Effective batch size | 16 |
| Learning rate | 1e-4 |
| LR schedule | cosine, 3% warmup |
| Optimizer | AdamW (`Trainer` default) |
| Total steps | 5,000 |
| Total training time | ~3h 3m (11,012s) |
| Throughput | 7.26 samples/sec |

Checkpoints were saved locally every 500 steps and mirrored to this Hub repo roughly every hour during training via a custom `TrainerCallback` (each upload replaced the repo's prior contents), with a final clean upload of just the adapter + processor after training completed.

### Training / Validation Loss

| Step | Training Loss | Validation Loss |
|---:|---:|---:|
| 500  | 0.1948 | 0.1918 |
| 1000 | 0.1842 | 0.1871 |
| 1500 | 0.1878 | 0.1858 |
| 2000 | 0.1827 | 0.1865 |
| 2500 | 0.1685 | 0.1856 |
| 3000 | 0.1809 | 0.1841 |
| 3500 | 0.1825 | 0.1837 |
| 4000 | 0.1730 | 0.1836 |
| 4500 | 0.1758 | 0.1833 |
| 5000 | 0.1860 | 0.1833 |

Final train loss: **0.2363** (mean over all logged steps, includes early-training values before it stabilized). Validation loss plateaued around **0.183** from step ~3500 onward, suggesting the model converged with room left in the schedule rather than overfitting.

## Usage

```python
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq
from peft import PeftModel

BASE_MODEL = "HuggingFaceTB/SmolVLM-256M-Instruct"
ADAPTER_REPO = "w4ashabii/SmolVLM256M_CropDisease"

processor = AutoProcessor.from_pretrained(ADAPTER_REPO)
base_model = AutoModelForVision2Seq.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO).to("cuda")
model.eval()

image = Image.open("your_crop_photo.jpg").convert("RGB")
question = "What crop is shown in this image, and does it have any disease? If so, name the disease."

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": question}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to("cuda")

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=64)

answer = processor.batch_decode(
    generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0]
print(answer.strip())
```

### With Unsloth (faster inference load)

```python
from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained("w4ashabii/SmolVLM256M_CropDisease")
FastVisionModel.for_inference(model)
# ... same generate() call as above
```

## Intended Use

- Assistive identification of crop type and visible disease symptoms from photos, for agricultural extension, research, or educational tooling focused on Nepali-grown crops.
- Not intended as a sole basis for treatment or pesticide decisions — outputs should be verified by an agronomist or local agricultural extension service, especially for high-stakes crop management decisions.

## Limitations

- Small base model (256M params) and LoRA-only tuning trade some accuracy for speed/size; expect a lighter-weight, less nuanced answer than larger VLMs.
- Loss computed over the full sequence (question + answer) rather than answer-only, so the model was also lightly trained to reproduce the (fixed) question text — this doesn't appear to have hurt convergence here but is a simplification versus masked/answer-only supervision.
- Coverage limited to the crops and disease classes present in `w4ashabii/nepali_crop_data`; performance on out-of-distribution crops, lighting conditions, or camera angles is untested.
- Not evaluated for robustness to adversarial or low-quality images (blur, occlusion, multiple crops in frame, etc.).

## Training Framework

Fine-tuned with 🤗 `transformers.Trainer` and LoRA adapters loaded/attached via [Unsloth](https://github.com/unslothai/unsloth)'s `FastVisionModel` (`use_gradient_checkpointing="unsloth"`) for reduced VRAM use and faster steps on a single T4 GPU.