[add] perf fig, vllm example

README.md

@@ -10,6 +10,11 @@ base_model:
 LLM trained to predict a character's emotional response in the given situation
 * Trained to predict in a structured output format.
 
+Prediction Performance:
+| Baseline (Qwen2.5-3B) | Trained |
+|:----------------------------------:|:------------------------------------:|
+| <img src="./assets/qwen2_5-3b-baseline_perf.png" alt="baseline_perf" width="90%" /> | <img src="./assets/finetuned_perf.png" alt="trained_perf" width="100%" /> |
+
 ## Quickstart
 The model is trained to predict in the following schema
 ```
@@ -66,6 +71,7 @@ prediction = generator(input_text)
 ```
 
 Using endpoint loaded with vllm & OpenAI client package
+* example of using vllm container is provided [here](./assets/run_vllm.sh)
 ```
 client = OpenAI(...)
 json_schema = EntryResult.model_json_schema()

assets/run_vllm.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+VLLM_VERSION="v0.8.4"
+
+BASE_MODEL_NAME="Qwen/Qwen2.5-3B-Instruct"
+
+ADAPTER_NAME="id4thomas/emotion-predictor-Qwen2.5-3B-Instruct"
+ADAPTER_DIR="..." # Path to the directory containing the LoRA adapter
+
+docker run --runtime nvidia --gpus all \
+        -v ${ADAPTER_DIR}:/vllm-workspace/adapter \
+        -v ./cache:/root/.cache/huggingface \
+        -p 8010:8000 \
+        --ipc=host \
+        vllm/vllm-openai:${VLLM_VERSION} \
+        --model "Qwen/Qwen2.5-3B-Instruct" \
+        --lora-modules '{"name": "${ADAPTER_NAME}", "path": "/vllm-workspace/adapter", "base_model_name": "${BASE_MODEL_NAME}"}' \
+        --enable-lora \
+        --max-lora-rank 16 \
+        --served-model-name ${BASE_MODEL_NAME} \
+        --gpu-memory-utilization=0.5