---
base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: peft
tags:
- llama-3
- qlora
- peft
- medical
- medquad
- fine-tuning
- 4bit
license: cc-by-nc-4.0
---

# 🧬 Llama-3.1-8B-MedQuAD-LoRA

**Llama-3.1-8B-MedQuAD-LoRA** is a parameter-efficient fine-tuned version of `meta-llama/Llama-3.1-8B-Instruct`, trained on the **MedQuAD (Medical Question Answering Dataset)** using **QLoRA** for factual and educational biomedical question answering.  


---

## ⚙️ Technical Overview

| Setting | Value |
|----------|--------|
| **Base model** | meta-llama/Llama-3.1-8B-Instruct |
| **Fine-tuning method** | QLoRA (4-bit NF4 quantization) |
| **LoRA rank / alpha** | r = 8 / α = 16 |
| **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| **Dropout** | 0.1 |
| **Precision** | FP16 mixed |
| **Batch size (effective)** | 8 (2 × grad accum = 4) |
| **Learning rate** | 2e-4 |
| **Optimizer** | AdamW (β₁ = 0.9, β₂ = 0.95) |
| **Scheduler** | Linear decay, warmup = 3 % |
| **Max sequence length** | 1024 |
| **Epochs** | 2 |
| **GPU** | NVIDIA T4 (15 GB VRAM) |
| **Training time** | ≈ 8 hours |
| **Final loss (train/val)** | 0.76 / 0.75 |

---