Update README.md

README.md

@@ -1,58 +1,333 @@
 ---
+license: llama3.2
 base_model: meta-llama/Llama-3.2-3B-Instruct
-library_name: transformers
-model_name: Llama-3.2-3B-Instruct-Medical-Finetune-v5
+library_name: peft
+pipeline_tag: text-generation
+language:
+  - en
 tags:
-- generated_from_trainer
-- trl
-- sft
-licence: license
+  - medical
+  - healthcare
+  - clinical
+  - clinical-decision-support
+  - question-answering
+  - medical-qa
+  - llama
+  - llama-3.2
+  - lora
+  - qlora
+  - peft
+  - parameter-efficient-fine-tuning
+  - 4-bit
+datasets:
+  - MohamedAhmedAE/Med_LLaMa3_fine-tuning_dataset
+  - medalpaca/medical_meadow_medqa
+  - medalpaca/medical_meadow_medical_flashcards
+  - medalpaca/medical_meadow_wikidoc
+  - medalpaca/medical_meadow_wikidoc_patient_information
+  - medalpaca/medical_meadow_cord19
+  - medalpaca/medical_meadow_pubmed_causal
+  - openlifescienceai/medmcqa
+  - bigbio/med_qa
+  - qiaojin/PubMedQA
+  - deepset/covid_qa_deepset
 ---
 
-# Model Card for Llama-3.2-3B-Instruct-Medical-Finetune-v5
+# Med-LLaMA3.2-3B — Medical QLoRA Adapter (LoRA weights only)
 
-This model is a fine-tuned version of [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+> Parameter-efficient medical adaptation of **Llama-3.2-3B** using **QLoRA** (4-bit NF4 + LoRA).
+> This repository contains the **LoRA adapter only** — it must be applied on top of the base
+> model at load time. For a ready-to-use, standalone checkpoint, see the **merged** version
+> linked below.
 
-## Quick start
+This is the **3B (balanced / mid-tier)** member of the **Med-LLaMA3** family introduced in the paper
+*“Med-LLaMA3: Advancing Medical Question-Answering Through Parameter-Efficient Fine-Tuning of Large
+Language Models”* (Applied Sciences, 2026). The family adapts the LLaMA-3 architecture to the medical
+domain by training only a small fraction of the base model’s parameters (**5.70% for this 3B variant**),
+achieving strong medical question-answering performance while reducing memory use by roughly **75%** via
+4-bit quantization — enabling development and inference on low-cost, consumer-grade hardware.
+
+The 3B variant offers a **balanced trade-off between computational efficiency and capacity** — a
+competitive mid-tier option with meaningfully better accuracy than the 1B and lower compute than the 8B.
+
+- 📄 **Paper:** [Med-LLaMA3 (Applied Sciences 2026, 16(12), 6158)](https://www.mdpi.com/2076-3417/16/12/6158) · DOI: [10.3390/app16126158](https://doi.org/10.3390/app16126158)
+- 💻 **Code:** [github.com/Mohamed-Ahmed-Abo-El-Enen/MasterPapers](https://github.com/Mohamed-Ahmed-Abo-El-Enen/MasterPapers)
+- 🧩 **Merged (standalone) version:** [`MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned-merged`](https://huggingface.co/MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned-merged)
+
+---
+
+## Model details
+
+| | |
+|---|---|
+| **Base model** | [`meta-llama/Llama-3.2-3B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) |
+| **Adaptation method** | QLoRA — 4-bit **NF4** quantization (double quantization) + LoRA |
+| **LoRA rank (`r`)** | 128 |
+| **LoRA alpha (`α`)** | 256 (scaling `α/r = 2.0`) |
+| **LoRA target modules** | All linear layers — `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
+| **Trainable parameters** | 194.51 M LoRA parameters = **5.70%** of the 3.40 B total (base weights frozen) |
+| **Compute dtype** | bfloat16 (mixed precision) |
+| **Architecture** | 28 decoder layers · hidden size 3072 · intermediate size 8192 · GQA (24 query heads, 8 KV heads) |
+| **Context window** | 128K tokens (inherited from base) |
+| **Vocabulary** | 128,256 tokens |
+| **Language** | English |
+| **License** | [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE) |
+
+> **ℹ️ Base checkpoint.** A LoRA adapter only loads correctly onto the *exact* base model it was trained
+> on. This adapter targets the **instruct** checkpoint `meta-llama/Llama-3.2-3B-Instruct` (consistent
+> with the released fine-tuned model). Use that same base in the code below.
+
+---
+
+## Intended uses
+
+**Primary use cases**
+
+- Medical **question answering** (multiple-choice and open-ended).
+- Clinical knowledge lookup and **clinical decision support** assistance.
+- A balanced mid-tier option when the 1B is too small and the 8B is too heavy.
+- A research baseline for parameter-efficient fine-tuning of small LLaMA models in healthcare.
+
+**Out of scope / not intended for**
+
+- Autonomous clinical decision-making or direct patient care without a qualified clinician in the loop.
+- Generating definitive diagnoses, prescriptions, or treatment plans.
+- Use as a substitute for professional medical advice, emergency services, or licensed care.
+
+See **[Limitations & responsible use](#limitations--responsible-use)** before any applied use.
+
+---
+
+## How to use
+
+This is a PEFT/LoRA adapter, so you load the **base model first** and then attach the adapter.
+
+```bash
+pip install -U transformers peft accelerate bitsandbytes torch
+```
+
+### Option A — 4-bit inference (lower GPU memory)
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+from peft import PeftModel
+
+BASE_MODEL = "meta-llama/Llama-3.2-3B-Instruct"
+ADAPTER    = "MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned"  # <-- this adapter repo
+
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
+base = AutoModelForCausalLM.from_pretrained(
+    BASE_MODEL,
+    quantization_config=bnb_config,
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(base, ADAPTER)
+model.eval()
+
+messages = [
+    {"role": "system", "content": "You are a knowledgeable medical assistant. Answer accurately and concisely."},
+    {"role": "user", "content": "What is the first-line treatment for uncomplicated community-acquired pneumonia in a healthy adult?"},
+]
+inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    out = model.generate(inputs, max_new_tokens=256, do_sample=False, temperature=0.0)
+print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+### Option B — full-precision inference
 
 ```python
-from transformers import pipeline
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+
+BASE_MODEL = "meta-llama/Llama-3.2-3B-Instruct"
+ADAPTER    = "MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned"
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetune-v5", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
+base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.bfloat16, device_map="auto")
+model = PeftModel.from_pretrained(base, ADAPTER)
 ```
 
+### Optional — merge the adapter into the base
+
+If you want a single standalone model (no PEFT dependency at inference), merge the weights:
+
+```python
+merged = model.merge_and_unload()
+merged.save_pretrained("Med-LLaMA3.2-3B-Medical-merged")
+tokenizer.save_pretrained("Med-LLaMA3.2-3B-Medical-merged")
+```
+
+*(A pre-merged checkpoint is also published separately — see the link at the top of this card.)*
+
+---
+
+## Training data
+
+The Med-LLaMA3 family was fine-tuned on a curated **medical instruction dataset of over 1.5 million
+samples**, organized along a three-axis taxonomy: **source type** (examination QA, clinical dialogue,
+biomedical literature, encyclopedic reference) × **clinical granularity** (basic science, clinical
+reasoning, patient communication) × **task format** (multiple-choice, open-ended QA, generative
+dialogue). All sources were consolidated into a unified instruction–response schema
+(`system`, `context`, `question`, `answer`, `choices`).
+
+Sources include:
+
+- **MedAlpaca / Medical Meadow** collection — MEDIQA, Medical Flashcards, WikiDoc, WikiDoc Patient
+  Information, MedQA, CORD-19, and PubMed Causal subsets
+- **MedMCQA** — Indian medical entrance exam (AIIMS & NEET PG) multiple-choice questions
+- **MedQA-USMLE** — USMLE-style 4-option multiple-choice questions (English)
+- **BigBIO MedQA** — standardized biomedical QA
+- **PubMedQA** — research questions over PubMed abstracts (yes/no/maybe)
+- **COVID-QA (deepset)** — COVID-19 / SARS-CoV-2 question answering
+- **MedQuAD** — consumer-health QA compiled from authoritative NIH sources
+- **HealthCareMagic** — real-world patient–doctor conversation transcripts
+
+The data-cleaning and corpus-assembly scripts are released in the
+[code repository](https://github.com/Mohamed-Ahmed-Abo-El-Enen/MasterPapers), and the final compiled
+fine-tuning dataset is available at
+[`MohamedAhmedAE/Med_LLaMa3_fine-tuning_dataset`](https://huggingface.co/datasets/MohamedAhmedAE/Med_LLaMa3_fine-tuning_dataset).
+
+> **Evaluation integrity:** The eight **MMLU medical subsets** were used **only for held-out
+> evaluation** and were **excluded** from the fine-tuning corpus. For benchmarks with official splits
+> (MedMCQA, MedQA-USMLE, PubMedQA), only the official **training** partitions were used for fine-tuning.
+
+---
+
 ## Training procedure
 
-[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/mohamed-ahmed/Llama-3.2-3B-Instruct-Medical-Finetune-v5/runs/44aemw1g) 
+LoRA and optimization settings are identical across the 1B, 3B, and 8B variants; sequence length, batch
+size, and gradient accumulation are scaled to each model’s memory footprint. The settings below are for
+the **3B** variant.
 
+| Setting | Value (3B) |
+|---|---|
+| Method | QLoRA (4-bit NF4 base, LoRA adapters in higher precision) |
+| LoRA `r` / `α` / dropout / bias | 128 / 256 / 0.05 / none |
+| Target modules | All linear layers (q, k, v, o, gate, up, down) |
+| Trainable params | 194.51 M (5.70% of 3.40 B) |
+| Quantization | 4-bit NF4 with double quantization (bitsandbytes) |
+| Optimizer | Paged AdamW 8-bit (β₁ = 0.9, β₂ = 0.999), weight decay 0.1 |
+| Learning rate / schedule | 2.0 × 10⁻⁵ / cosine annealing, 5 warmup steps |
+| Epochs | 5 |
+| Max sequence length | 2048 |
+| Batch size / grad accumulation | 4 per device / 16 steps |
+| Max gradient norm | 1.0 |
+| Precision & memory | bfloat16 · gradient checkpointing · DeepSpeed ZeRO-2 · FlashAttention-2 |
+| Hardware | 2 × NVIDIA RTX 4050 (12 GB), ~30 days |
+| Experiment tracking | Weights & Biases |
 
-This model was trained with SFT.
+The QLoRA recipe keeps the base weights frozen and quantized, allocating optimizer state only for the
+LoRA parameters — which is what makes fine-tuning feasible on consumer hardware.
+
+---
 
-### Framework versions
+## Evaluation
 
-- TRL: 0.21.0
-- Transformers: 4.55.4
-- Pytorch: 2.8.0
-- Datasets: 3.6.0
-- Tokenizers: 0.21.4
+Evaluation in the paper uses the **EleutherAI LM Evaluation Harness** with **5-shot** prompting on the
+eight MMLU medical subsets (Anatomy, Clinical Knowledge, College Biology, College Medicine, Medical
+Genetics, Nutrition, Professional Medicine, Virology). Reported comparisons include **McNemar’s test**
+p-values and **95% bootstrap confidence intervals**.
 
-## Citations
+The table below reports the 3B model’s 5-shot accuracy (%) on each MMLU medical subset, with 95%
+bootstrap confidence intervals (1000 resamples), as published in Table 7 of the paper. For context, the
+family’s mean accuracy scales with model size: **1B = 48.64%**, **3B = 64.24%**, **8B = 75.71%**.
 
+| MMLU medical subset (5-shot) | Med-LLaMA3.2-3B (acc. %) |
+|---|---|
+| Anatomy | 59.52 (±4.26) |
+| Clinical Knowledge | 68.17 (±2.89) |
+| College Biology | 71.53 (±3.77) |
+| College Medicine | 57.65 (±3.78) |
+| Medical Genetics | 75.00 (±4.35) |
+| Nutrition | 68.32 (±2.69) |
+| Professional Medicine | 70.59 (±2.77) |
+| Virology | 43.17 (±3.84) |
+| **Mean (8 subsets)** | **64.24** |
+
+> The paper reports an untuned baseline only for the 8B model (vs. `Llama-3.1-8B-Instruct`); it does
+> **not** include an untuned `Llama-3.2-3B` baseline on these subsets. See Table 7 of the paper for the
+> full cross-model comparison (1B, 8B, and other ≤8B models) with statistical tests.
+
+See the [paper](https://www.mdpi.com/2076-3417/16/12/6158) for full tables, statistical tests, and
+confidence intervals.
+
+---
 
+## Limitations & responsible use
+
+- **Not a medical device.** This model is a research artifact. It must **not** be used for autonomous
+  diagnosis, treatment, prescribing, or any decision affecting patient care without review by a
+  qualified healthcare professional.
+- **Hallucination risk.** Like all LLMs, it can produce fluent but incorrect or fabricated medical
+  information. Always verify outputs against authoritative sources.
+- **Mid-tier capacity.** The 3B is a balanced variant; for the highest accuracy on complex clinical
+  reasoning, prefer the 8B variant when resources allow. For the smallest footprint, the 1B is available.
+- **Abbreviation ambiguity.** Medical abbreviations are a known error source. The paper’s safety pilot
+  shows that **context-disambiguation preprocessing** reduces the highest-severity abbreviation
+  errors (from 30% to 10% on a held-out set); consider applying similar preprocessing.
+- **Data & bias.** Training data may under-represent certain populations, conditions, or regional
+  practices, and may encode biases present in the source corpora.
+- **Privacy & compliance.** Do not input protected health information (PHI) unless your deployment is
+  appropriately secured and compliant with applicable regulations (e.g., HIPAA, GDPR).
+- **English only.** Performance outside English is not evaluated.
+
+---
+
+## License
+
+This adapter is released under the **[Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE)**,
+inherited from the base model. By using it you agree to Meta’s Llama 3.2 license terms and
+Acceptable Use Policy. Review the licenses of the individual training datasets for any additional
+restrictions on derived use.
+
+---
+
+## Citation
+
+If you use this model, please cite the paper:
 
-Cite TRL as:
-    
 ```bibtex
-@misc{vonwerra2022trl,
-	title        = {{TRL: Transformer Reinforcement Learning}},
-	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
-	year         = 2020,
-	journal      = {GitHub repository},
-	publisher    = {GitHub},
-	howpublished = {\url{https://github.com/huggingface/trl}}
+@article{aboelenen2026medllama3,
+  title   = {Med-LLaMA3: Advancing Medical Question-Answering Through Parameter-Efficient Fine-Tuning of Large Language Models},
+  author  = {Abo El-Enen, Mohamed Ahmed and Ismail, Sally S. and Nazmy, Taymoor Mohamed},
+  journal = {Applied Sciences},
+  volume  = {16},
+  number  = {12},
+  pages   = {6158},
+  year    = {2026},
+  publisher = {MDPI},
+  doi     = {10.3390/app16126158},
+  url     = {https://www.mdpi.com/2076-3417/16/12/6158}
 }
-```
+```
+
+## Authors & contact
+
+Mohamed Ahmed Abo El-Enen, Sally S. Ismail, and Taymoor Mohamed Nazmy
+Faculty of Computer and Information Sciences, Ain Shams University, Cairo, Egypt.
+
+---
+
+## Model family
+
+| Variant | Type | Repository |
+|---|---|---|
+| Med-LLaMA3.2-1B | Adapter | [`MohamedAhmedAE/Llama-3.2-1B-Instruct-Medical-Finetuned`](https://huggingface.co/MohamedAhmedAE/Llama-3.2-1B-Instruct-Medical-Finetuned) |
+| Med-LLaMA3.2-1B | Merged | [`MohamedAhmedAE/Llama-3.2-1B-Instruct-Medical-Finetuned-merged`](https://huggingface.co/MohamedAhmedAE/Llama-3.2-1B-Instruct-Medical-Finetuned-merged) |
+| **Med-LLaMA3.2-3B** | **Adapter** | **this repo** — [`MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned`](https://huggingface.co/MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned) |
+| Med-LLaMA3.2-3B | Merged | [`MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned-merged`](https://huggingface.co/MohamedAhmedAE/Llama-3.2-3B-Instruct-Medical-Finetuned-merged) |
+| Med-LLaMA3.1-8B | Adapter | [`MohamedAhmedAE/Llama-3.1-8B-Instruct-Medical-Finetuned`](https://huggingface.co/MohamedAhmedAE/Llama-3.1-8B-Instruct-Medical-Finetuned) |
+| Med-LLaMA3.1-8B | Merged | [`MohamedAhmedAE/Llama-3.1-8B-Instruct-Medical-Finetuned-merged`](https://huggingface.co/MohamedAhmedAE/Llama-3.1-8B-Instruct-Medical-Finetuned-merged) |
+
+**Fine-tuning dataset:** [`MohamedAhmedAE/Med_LLaMa3_fine-tuning_dataset`](https://huggingface.co/datasets/MohamedAhmedAE/Med_LLaMa3_fine-tuning_dataset)