File size: 4,596 Bytes
3e114a3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | ---
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- medical
- healthcare
- qlora
- lora
- qwen
- fine-tuned
- sft
- transformers
- trl
datasets:
- lavita/medical-qa-datasets
---
# ClinicalQwen-1.5B-Medical
A QLoRA fine-tuned version of [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) for medical question answering.
> **Disclaimer**: This model is for educational and research purposes only. Do not use it as a substitute for professional medical advice, diagnosis, or treatment. Always consult a qualified healthcare professional.
## How to Use
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
# Load base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
base_model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-1.5B-Instruct",
quantization_config=bnb_config,
device_map="auto",
)
# Load fine-tuned adapter
model = PeftModel.from_pretrained(base_model, "Coddieharsh/ClinicalQwen-1.5B-Medical")
tokenizer = AutoTokenizer.from_pretrained("Coddieharsh/ClinicalQwen-1.5B-Medical")
# Generate response
def ask(question):
prompt = (
"<|im_start|>system\n"
"You are a knowledgeable medical assistant. Provide accurate, evidence-based "
"medical information. Always recommend consulting a healthcare professional "
"for personal medical advice.<|im_end|>\n"
f"<|im_start|>user\n{question}<|im_end|>\n"
"<|im_start|>assistant\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.7,
do_sample=True,
top_p=0.9,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
)
response = tokenizer.decode(outputs[0], skip_special_tokens=False)
return response.split("<|im_start|>assistant")[-1].split("<|im_end|>")[0].strip()
print(ask("What are the symptoms of Type 2 diabetes?"))
```
## Training Details
| Parameter | Value |
|-----------|-------|
| Base Model | Qwen/Qwen2.5-1.5B-Instruct |
| Fine-tuning Method | QLoRA (4-bit NF4) |
| Dataset | lavita/medical-qa-datasets |
| Training Samples | 5,000 |
| Epochs | 1 |
| Final Loss | 1.8991 |
| Training Time | ~2.4 hours |
| Hardware | Google Colab T4 (16GB) |
| LoRA Rank (r) | 8 |
| LoRA Alpha | 16 |
| Learning Rate | 2e-4 |
| Batch Size (effective) | 16 (2 × 8 grad accum) |
| Optimizer | paged_adamw_8bit |
| Scheduler | Cosine |
| Max Sequence Length | 512 |
| Trainable Parameters | 9.2M / 1,552M (< 1%) |
**Target modules**: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
## Evaluation Results
Evaluated on 10 medical benchmark questions:
| Metric | Score | Note |
|--------|-------|------|
| BERTScore F1 | 0.5913 | Semantic similarity (primary metric) |
| Perplexity | 3.14 | Fluency — lower is better |
| Keyword Coverage | 86.0% | Medical term coverage |
| ROUGE-L | 0.1219 | Surface overlap (low = paraphrasing, expected) |
| BLEU | 0.0099 | N-gram precision (low = paraphrasing, expected) |
Low BLEU/ROUGE is expected for generative medical assistants — the model paraphrases rather than copying reference text verbatim.
## Sample Output
**Q:** What are the symptoms of Type 2 diabetes?
**A:** Type 2 diabetes is characterized by high blood sugar levels over time. Common symptoms include:
- Frequent urination (polyuria)
- Increased thirst (polydipsia)
- Fatigue and low energy
- Blurred vision
- Slow-healing wounds
- Numbness or tingling in hands and feet
If you experience these symptoms, consult a healthcare professional for proper diagnosis.
## Limitations
- Trained on 5,000 samples (1 epoch) — a larger dataset and more epochs would improve accuracy
- Not validated by medical professionals — outputs may contain inaccuracies
- Should not be used for clinical decision-making
- Best suited for general health information queries
## Repository
Full training code, notebooks, and evaluation scripts: [github.com/HarshTomar1234/ClinicalQwen](https://github.com/HarshTomar1234/ClinicalQwen)
|