Instructions to use valeriojob/MedGPT-Gemma2-9B-BA-v.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use valeriojob/MedGPT-Gemma2-9B-BA-v.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="valeriojob/MedGPT-Gemma2-9B-BA-v.1")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("valeriojob/MedGPT-Gemma2-9B-BA-v.1")
model = AutoModelForCausalLM.from_pretrained("valeriojob/MedGPT-Gemma2-9B-BA-v.1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use valeriojob/MedGPT-Gemma2-9B-BA-v.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "valeriojob/MedGPT-Gemma2-9B-BA-v.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "valeriojob/MedGPT-Gemma2-9B-BA-v.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/valeriojob/MedGPT-Gemma2-9B-BA-v.1

SGLang

How to use valeriojob/MedGPT-Gemma2-9B-BA-v.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "valeriojob/MedGPT-Gemma2-9B-BA-v.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "valeriojob/MedGPT-Gemma2-9B-BA-v.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "valeriojob/MedGPT-Gemma2-9B-BA-v.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "valeriojob/MedGPT-Gemma2-9B-BA-v.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Unsloth Desktop
Docker Model Runner
How to use valeriojob/MedGPT-Gemma2-9B-BA-v.1 with Docker Model Runner:
```
docker model run hf.co/valeriojob/MedGPT-Gemma2-9B-BA-v.1
```

MedGPT-Gemma2-9B-BA-v.1 / README.md

valeriojob

Update README.md

39677fd verified about 2 years ago

preview code

Raw

History Blame Contribute Delete

2.29 kB

	---
	base_model: unsloth/gemma-2-9b
	language:
	- en
	license: apache-2.0
	tags:
	- text-generation-inference
	- transformers
	- unsloth
	- llama
	- trl
	- sft
	---

	# MedGPT-Gemma2-9B-v.1

	- This model is a fine-tuned version of [unsloth/gemma-2-9b](https://huggingface.co/unsloth/gemma-2-9b) on an dataset created by [Valerio Job](https://huggingface.co/valeriojob) together with GPs based on real medical data.
	- Version 1 (v.1) of MedGPT is the very first version of MedGPT and the training dataset has been kept simple and small with only 60 examples.
	- This repo includes the 16bit format of the model as well as the LoRA adapters of the model. There is a separate repo called [valeriojob/MedGPT-Gemma2-9B-BA-v.1-GGUF](https://huggingface.co/valeriojob/MedGPT-Gemma2-9B-BA-v.1-GGUF) that includes the quantized versions of this model in GGUF format.
	- This model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

	## Model description

	This model acts as a supplementary assistance to GPs helping them in medical and admin tasks.

	## Intended uses & limitations

	The fine-tuned model should not be used in production! This model has been created as a initial prototype in the context of a bachelor thesis.

	## Training and evaluation data

	The dataset (train and test) used for fine-tuning this model can be found here: [datasets/valeriojob/BA-v.1](https://huggingface.co/datasets/valeriojob/BA-v.1)

	## Training procedure

	### Training hyperparameters

	The following hyperparameters were used during training:
	- per_device_train_batch_size = 2,
	- gradient_accumulation_steps = 4,
	- warmup_steps = 5,
	- max_steps = 60,
	- learning_rate = 2e-4,
	- fp16 = not is_bfloat16_supported(),
	- bf16 = is_bfloat16_supported(),
	- logging_steps = 1,
	- optim = "adamw_8bit",
	- weight_decay = 0.01,
	- lr_scheduler_type = "linear",
	- seed = 3407,
	- output_dir = "outputs"

	### Training results

	\| Training Loss \| Step \|
	\|:-------------:\|:----:\|
	\| 2.237900 \| 1 \|
	\| 2.292200 \| 2 \|
	\| 2.215200 \| 3 \|
	\| 1.561200 \| 5 \|
	\| 0.584500 \| 10 \|
	\| 0.372500 \| 15 \|
	\| 0.258600 \| 20 \|
	\| 0.126300 \| 30 \|
	\| 0.064100 \| 40 \|
	\| 0.040800 \| 50 \|
	\| 0.045700 \| 60 \|

	## Licenses
	- License: apache-2.0