---
base_model: microsoft/Florence-2-base
repository: francipaolo/florence-2-pal-comp-v1
---

# Florence-2 Finetuned for Medieval Illustration Description

This model is a finetuned version of the Microsoft Florence-2-base model, specifically trained to generate detailed descriptions of medieval illustrations based on provided image and text data.

## Model Description

The base model, Florence-2, is a versatile visual language model. This finetuned version has been trained on a dataset of medieval illustrations and their associated descriptive terms. The goal of the finetuning was to improve the model's ability to generate accurate and relevant captions for this specific domain.

## Training Data

The model was finetuned using a dataset containing images of medieval illustrations and structured textual descriptions (terms). The dataset was processed to normalize the terms and prepare the data for training. 

IMPORTANT: The caption used for training where only key terms from the [Timel thesaurus](https://datu.ehess.fr/timel/fr/), translated from french into english using the LLM model _Mistral-Small-24B-Instruct-2501_.

## Usage

This model can be used for generating descriptions of medieval illustrations.

```python
python from transformers import AutoModelForCausalLM, AutoProcessor import torch from PIL import Image

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

#loading model and processor
AutoModelForCausalLM.from_pretrained("francipaolo/florence-2-pal-comp-v1", trust_remote_code=True).to(device) 
processor = AutoProcessor.from_pretrained("francipaolo/florence-2-pal-comp-v1", trust_remote_code=True)

#defining function
def run_example(task_prompt, text_input, image):
    if text_input is None:
        prompt = task_prompt
    else:
        prompt = task_prompt + text_input

    # Ensure the image is in RGB mode
    if image.mode != "RGB":
        image = image.convert("RGB")

    inputs = processor(text=prompt, images=image, return_tensors="pt").to(device)
    generated_ids = model.generate(
        input_ids=inputs["input_ids"],
        pixel_values=inputs["pixel_values"],
        max_new_tokens=1024,
        num_beams=3
    )
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
    parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=(image.width, image.height))
    return parsed_answer


for idx in range(2):
    image = data['train'][idx]['image'] #pass any PIL Image object (converted to RGB) 
    description = run_example('<MORE_DETAILED_CAPTION>', '', image)
    print(f"Generated Description: {description}")
    display(image.resize([350, 350]))
```