metadata
base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: peft
tags:
- llama-3
- qlora
- peft
- medical
- medquad
- fine-tuning
- 4bit
license: cc-by-nc-4.0
🧬 Llama-3.1-8B-MedQuAD-LoRA
Llama-3.1-8B-MedQuAD-LoRA is a parameter-efficient fine-tuned version of meta-llama/Llama-3.1-8B-Instruct, trained on the MedQuAD (Medical Question Answering Dataset) using QLoRA for factual and educational biomedical question answering.
⚙️ Technical Overview
| Setting | Value |
|---|---|
| Base model | meta-llama/Llama-3.1-8B-Instruct |
| Fine-tuning method | QLoRA (4-bit NF4 quantization) |
| LoRA rank / alpha | r = 8 / α = 16 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Dropout | 0.1 |
| Precision | FP16 mixed |
| Batch size (effective) | 8 (2 × grad accum = 4) |
| Learning rate | 2e-4 |
| Optimizer | AdamW (β₁ = 0.9, β₂ = 0.95) |
| Scheduler | Linear decay, warmup = 3 % |
| Max sequence length | 1024 |
| Epochs | 2 |
| GPU | NVIDIA T4 (15 GB VRAM) |
| Training time | ≈ 8 hours |
| Final loss (train/val) | 0.76 / 0.75 |