| --- |
| base_model: meta-llama/Llama-3.1-8B-Instruct |
| library_name: peft |
| tags: |
| - llama-3 |
| - qlora |
| - peft |
| - medical |
| - medquad |
| - fine-tuning |
| - 4bit |
| license: cc-by-nc-4.0 |
| --- |
| |
| # 🧬 Llama-3.1-8B-MedQuAD-LoRA |
|
|
| **Llama-3.1-8B-MedQuAD-LoRA** is a parameter-efficient fine-tuned version of `meta-llama/Llama-3.1-8B-Instruct`, trained on the **MedQuAD (Medical Question Answering Dataset)** using **QLoRA** for factual and educational biomedical question answering. |
|
|
|
|
| --- |
|
|
| ## ⚙️ Technical Overview |
|
|
| | Setting | Value | |
| |----------|--------| |
| | **Base model** | meta-llama/Llama-3.1-8B-Instruct | |
| | **Fine-tuning method** | QLoRA (4-bit NF4 quantization) | |
| | **LoRA rank / alpha** | r = 8 / α = 16 | |
| | **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | |
| | **Dropout** | 0.1 | |
| | **Precision** | FP16 mixed | |
| | **Batch size (effective)** | 8 (2 × grad accum = 4) | |
| | **Learning rate** | 2e-4 | |
| | **Optimizer** | AdamW (β₁ = 0.9, β₂ = 0.95) | |
| | **Scheduler** | Linear decay, warmup = 3 % | |
| | **Max sequence length** | 1024 | |
| | **Epochs** | 2 | |
| | **GPU** | NVIDIA T4 (15 GB VRAM) | |
| | **Training time** | ≈ 8 hours | |
| | **Final loss (train/val)** | 0.76 / 0.75 | |
| |
| --- |
