README.md

# Enhanced Explanations for Kvasir-VQA

This repository contains our process for generating **textual** and **visual explanations** on top of the original **SimulaMet/Kvasir-VQA-x1** dataset. The work enhances standard VQA answers with grounded reasoning, clinical language, and region-linked visual cues.  

---

## Textual Explanation Augmentation

We extended the original **SimulaMet/Kvasir-VQA-x1** dataset with additional signals:

- **Natural VQA answers** from *SimulaMet/Kvasir-VQA-x1*.  
- **Ground-truth explanations** from *SimulaMet-HOST/Kvasir-VQA*.  
- **Visual descriptions** generated by **Gemma 27B**, which captured contextual details of the images.  

By combining these three sources for each image and question pair, we created **enhanced explanations** grounded in both natural responses and domain-specific cues.  
![Figure 1: Textual Explanation Overview](Textual_Exp.png)

---

## Visual Explanation Augmentation

To complement textual reasoning, we linked **region-based visual cues** to answers:

- Used **pseudo masks** generated via prompt-guided segmentation (e.g., ClipSeg).  
- Integrated **existing polyp and instrument masks** from *Kvasir-SEG*.  
- Linked masks to related answers using metadata from *SimulaMet/Kvasir-VQA-x1*.  

This allowed the model to ground its predictions in specific image regions (e.g., polyps, instruments, anatomical landmarks).  
![Figure 2: Visual Grounding Pipeline](Visual_Grnd.png)

---

## Training Details

We trained the **Florence-2** model with LoRA fine-tuning:

- **LoRA config**: `r=128`, `a=256`  
- **Tokens used**:
  - `<MedVQA> {question}` → Standard VQA task  
  - `<MedVQA_EXPLAIN> {question} Explain in Detail` → Textual explanation task  
  - `<REFERRING_EXPRESSION_SEGMENTATION>` → Segmentation task (masks converted to Florence-supported location tokens)  

> ⚠️ Note: This version of the model is also trained on the test split.

---

## Caption-Based Post-Processing

In addition to VQA answers and explanations, we appended an **auto-generated caption** using the `<MORE_DETAILED_CAPTION>` token.  
- Interestingly, the model learned to produce **better grounded captions** after training, even though captioning was never explicitly part of the training objective.  
- These captions serve as a **natural clinical narrative** to enrich explanations.

---

## Example JSON Entry

Below is an example of the final output format combining all signals:

```json
{
    "val_id": 1313,
    "img_id": "cl8k2u1qk1ezn0832fzc5hr77",
    "question": "How many polyps are visible in the image?",
    "answer": "one polyp identified",
    "textual_explanation": "One polyp is identified in the image. It is a Paris Ip type polyp, measuring between 5-10mm in size, and appears as a rounded, pale pink mass with a slightly textured surface and small white spots scattered across its surface.\nOverall explaination of image: The image shows a single polypoid lesion in the gastrointestinal tract. The lesion appears pink in color and has a smooth, rounded shape with small white spots scattered across its surface. It is located in the center-right and lower-right regions of the image.",
    "visual_explanation": [
      {
        "type": "segmentation_mask",
        "data": "visuals/_mask_1313.jpg",
        "description": "Highlighted mask showing the region of interest supporting the answer."
      }
    ],
    "confidence_score": 0.9377866668833627
  }
```

## Example Mask for the above:
![Example Mask](_mask_1313.jpg)

---

## Confidence Calculation

For each generated explanation, we also estimate a **confidence score** based on the model’s decoding stability:  

- At every decoding step, we compute the **top-k probability mass** (sum of probabilities of the k most likely tokens).  
- This top-k mass reflects how concentrated the model’s belief is in its most likely continuations.  
- We average these values across all generated tokens to get the final **stability confidence score**.  

This score lies between 0 and 1, with higher values indicating that the model was consistently confident in its token predictions during explanation generation.


## Summary

- **Textual explanations** = Fusion of natural VQA, ground-truth HOST, and visual descriptions.  
- **Visual explanations** = Masks + segmentation linked to VQA metadata.  
- **Training** = Florence-2 with LoRA, multi-task prompting.  
- **Post-processing** = Appended auto-generated captions for better clinical context.