File size: 1,195 Bytes
501b45c ab6fc9c 501b45c ab6fc9c 501b45c ab6fc9c 501b45c ab6fc9c 501b45c ab6fc9c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | ---
base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: peft
tags:
- llama-3
- qlora
- peft
- medical
- medquad
- fine-tuning
- 4bit
license: cc-by-nc-4.0
---
# 🧬 Llama-3.1-8B-MedQuAD-LoRA
**Llama-3.1-8B-MedQuAD-LoRA** is a parameter-efficient fine-tuned version of `meta-llama/Llama-3.1-8B-Instruct`, trained on the **MedQuAD (Medical Question Answering Dataset)** using **QLoRA** for factual and educational biomedical question answering.
---
## ⚙️ Technical Overview
| Setting | Value |
|----------|--------|
| **Base model** | meta-llama/Llama-3.1-8B-Instruct |
| **Fine-tuning method** | QLoRA (4-bit NF4 quantization) |
| **LoRA rank / alpha** | r = 8 / α = 16 |
| **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| **Dropout** | 0.1 |
| **Precision** | FP16 mixed |
| **Batch size (effective)** | 8 (2 × grad accum = 4) |
| **Learning rate** | 2e-4 |
| **Optimizer** | AdamW (β₁ = 0.9, β₂ = 0.95) |
| **Scheduler** | Linear decay, warmup = 3 % |
| **Max sequence length** | 1024 |
| **Epochs** | 2 |
| **GPU** | NVIDIA T4 (15 GB VRAM) |
| **Training time** | ≈ 8 hours |
| **Final loss (train/val)** | 0.76 / 0.75 |
--- |