---
title: hyzhouMedVersa
sdk: gradio
sdk_version: 4.24.0
---
# MedVersa: A Generalist Learner for Multifaceted Medical Image Interpretation

A model implementation for our paper [A Generalist Learner for Multifaceted Medical Image Interpretation](https://arxiv.org/abs/2405.07988).

MedVersa is a compound medical AI system that coordinates multimodal inputs, orchestrates models and tools for various medical imaging tasks, and generates multimodal outputs.

## Installation

### Prerequisites

- [Python](https://www.python.org/)
- [NVIDIA CUDA-compatible GPU](https://developer.nvidia.com/cuda-gpus)
- [Miniconda](https://docs.anaconda.com/free/miniconda/index.html) or [Anaconda](https://www.anaconda.com/)

### Environment Setup

1. Create and activate the conda environment:

```bash
# For NVIDIA A100 GPUs
conda env create -f environment.yml
conda activate medversa

# For NVIDIA H100 GPUs (CUDA 11.8)
conda env create -f environment_cu118.yml
conda activate medversa
```

### Troubleshooting

If you encounter dependency issues, try these solutions:

```bash
# Fix OpenCV issues
pip install opencv-contrib-python

# Fix incompatible torchvision version
pip install torchvision==0.15.2+cu118 --index-url https://download.pytorch.org/whl/cu118
```

## Usage

### Basic Inference

```python
from utils import *
from torch import cuda

# Initialize model
device = 'cuda' if cuda.is_available() else 'cpu'
model_cls = registry.get_model_class('medomni') # medomni is the architecture name :)
model = model_cls.from_pretrained('hyzhou/MedVersa').to(device).eval()

# Define example input
example = {
    'images': ["./demo_ex/c536f749-2326f755-6a65f28f-469affd2-26392ce9.png"],
    'context': "Age:30-40.\nGender:F.\nIndication: ___-year-old female with end-stage renal disease not on dialysis presents with dyspnea. PICC line placement.\nComparison: None.",
    'prompt': "How would you characterize the findings from <img0>?",
    'modality': "cxr",
    'task': "report generation"
}

# Configure generation parameters
params = {
    'num_beams': 1,
    'do_sample': True,
    'min_length': 1,
    'top_p': 0.9,
    'repetition_penalty': 1,
    'length_penalty': 1,
    'temperature': 0.1
}

# Generate predictions
seg_mask_2d, seg_mask_3d, output_text = generate_predictions(
    model, 
    example['images'],
    example['context'],
    example['prompt'],
    example['modality'],
    example['task'],
    **params,
    device,
)
print(output_text)
```

For more detailed examples and usage scenarios, see `inference.py`.

## Prompts
More prompts can be found in `medomni/datasets/prompts.json`.