README.md

---
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- medical
- healthcare
- qlora
- lora
- qwen
- fine-tuned
- sft
- transformers
- trl
datasets:
- lavita/medical-qa-datasets
---

# ClinicalQwen-1.5B-Medical

A QLoRA fine-tuned version of [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) for medical question answering.

> **Disclaimer**: This model is for educational and research purposes only. Do not use it as a substitute for professional medical advice, diagnosis, or treatment. Always consult a qualified healthcare professional.

## How to Use

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Load fine-tuned adapter
model = PeftModel.from_pretrained(base_model, "Coddieharsh/ClinicalQwen-1.5B-Medical")
tokenizer = AutoTokenizer.from_pretrained("Coddieharsh/ClinicalQwen-1.5B-Medical")

# Generate response
def ask(question):
    prompt = (
        "<|im_start|>system\n"
        "You are a knowledgeable medical assistant. Provide accurate, evidence-based "
        "medical information. Always recommend consulting a healthcare professional "
        "for personal medical advice.<|im_end|>\n"
        f"<|im_start|>user\n{question}<|im_end|>\n"
        "<|im_start|>assistant\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            temperature=0.7,
            do_sample=True,
            top_p=0.9,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(outputs[0], skip_special_tokens=False)
    return response.split("<|im_start|>assistant")[-1].split("<|im_end|>")[0].strip()

print(ask("What are the symptoms of Type 2 diabetes?"))
```

## Training Details

| Parameter | Value |
|-----------|-------|
| Base Model | Qwen/Qwen2.5-1.5B-Instruct |
| Fine-tuning Method | QLoRA (4-bit NF4) |
| Dataset | lavita/medical-qa-datasets |
| Training Samples | 5,000 |
| Epochs | 1 |
| Final Loss | 1.8991 |
| Training Time | ~2.4 hours |
| Hardware | Google Colab T4 (16GB) |
| LoRA Rank (r) | 8 |
| LoRA Alpha | 16 |
| Learning Rate | 2e-4 |
| Batch Size (effective) | 16 (2 × 8 grad accum) |
| Optimizer | paged_adamw_8bit |
| Scheduler | Cosine |
| Max Sequence Length | 512 |
| Trainable Parameters | 9.2M / 1,552M (< 1%) |

**Target modules**: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`

## Evaluation Results

Evaluated on 10 medical benchmark questions:

| Metric | Score | Note |
|--------|-------|------|
| BERTScore F1 | 0.5913 | Semantic similarity (primary metric) |
| Perplexity | 3.14 | Fluency — lower is better |
| Keyword Coverage | 86.0% | Medical term coverage |
| ROUGE-L | 0.1219 | Surface overlap (low = paraphrasing, expected) |
| BLEU | 0.0099 | N-gram precision (low = paraphrasing, expected) |

Low BLEU/ROUGE is expected for generative medical assistants — the model paraphrases rather than copying reference text verbatim.

## Sample Output

**Q:** What are the symptoms of Type 2 diabetes?

**A:** Type 2 diabetes is characterized by high blood sugar levels over time. Common symptoms include:
- Frequent urination (polyuria)
- Increased thirst (polydipsia)
- Fatigue and low energy
- Blurred vision
- Slow-healing wounds
- Numbness or tingling in hands and feet

If you experience these symptoms, consult a healthcare professional for proper diagnosis.

## Limitations

- Trained on 5,000 samples (1 epoch) — a larger dataset and more epochs would improve accuracy
- Not validated by medical professionals — outputs may contain inaccuracies
- Should not be used for clinical decision-making
- Best suited for general health information queries

## Repository

Full training code, notebooks, and evaluation scripts: [github.com/HarshTomar1234/ClinicalQwen](https://github.com/HarshTomar1234/ClinicalQwen)