README.md

---
license: apache-2.0
language:
- en
base_model:
- google-t5/t5-large
library_name: peft
pipeline_tag: text2text-generation
datasets:
- rajpurkar/squad
tags:
- t5
- encoder-decoder
- question-answering
- extractive-qa
- squad
- lora
- peft
- parameter-efficient
metrics:
- exact_match
- f1
model-index:
- name: t5-large-lora-r8-squad
  results:
  - task:
      type: question-answering
      name: Extractive Question Answering
    dataset:
      name: SQuAD (validation, 256-example eval subset)
      type: rajpurkar/squad
      split: validation
    metrics:
    - type: exact_match
      value: 0.6445
      name: Exact Match
    - type: f1
      value: 0.8152
      name: Token F1
---

# T5-large — LoRA r=8 adapter on SQuAD

A **LoRA adapter** (PEFT) for `google-t5/t5-large`, trained for extractive QA on SQuAD.
Rank 8 on the `q` and `v` projections only → **~2.4M trainable params (0.32% of the model)**.

**TL;DR:** this 9.5 MB adapter reaches **F1 0.8152 / EM 0.6445** — statistically tied with
full fine-tuning (F1 0.8162) and with the cross-attention-only variant (F1 0.8128), and it
**beats parameter-matched GPT-2-large full fine-tune (F1 0.5041) by +0.31 F1** while training
0.32% of the weights versus its 100%. Architecture alignment, not the number of tuned
parameters, drives the result.

> This repo contains the **PEFT adapter only** (`adapter_model.safetensors`, ~9.5 MB).
> The base weights are pulled from `google-t5/t5-large` at load time.

## Results (SQuAD validation)

| Model | Architecture | Trainable / Total | EM | Token F1 |
| --- | --- | ---: | ---: | ---: |
| GPT-2-large | decoder-only | 774M / 774M | 0.3516 | 0.5041 |
| T5-large XA-only | enc-dec | 100.7M / 737M | 0.6406 | 0.8128 |
| **T5-large LoRA r=8 (this adapter)** | enc-dec | **2.4M / 740M** | **0.6445** | **0.8152** |
| T5-large full fine-tune | enc-dec | 737M / 737M | 0.6602 | 0.8162 |

Validation loss 0.3061, perplexity 1.36.

> Eval note: T5 numbers use a 256-example SQuAD-validation generation subset (4-beam search);
> GPT-2-large uses 512. Magnitudes are comparable and reproduce the paper's ordering.

## How to use

Trained with the input prefix `answer question: ` prepended to a
`question: ... context: ...` source string — match it exactly at inference:

```python
from transformers import T5ForConditionalGeneration, AutoTokenizer
from peft import PeftModel

base = "google-t5/t5-large"
adapter = "medelharchaoui/t5-large-lora-r8-squad"

tok = AutoTokenizer.from_pretrained(adapter)
model = T5ForConditionalGeneration.from_pretrained(base)
model = PeftModel.from_pretrained(model, adapter)
model = model.merge_and_unload()  # optional: fold LoRA into base for faster inference

question = "What culture do 'bairn' and 'hyem' originate from?"
context = ("'bairn' and 'hyem' are geordie words with origins in scandinavia; barn and hjem "
           "are the corresponding modern norwegian and danish words.")
text = f"answer question: question: {question} context: {context}"
ids = tok(text, return_tensors="pt", truncation=True, max_length=384).input_ids
print(tok.decode(model.generate(ids, num_beams=4, max_new_tokens=16)[0], skip_special_tokens=True))
```

## Training

| Setting | Value |
| --- | --- |
| Base model | `google-t5/t5-large` (737M) |
| LoRA | r=8, alpha=32, dropout=0.05, target modules `["q", "v"]` |
| Trainable params | ~2.4M (0.32%) |
| Dataset | `rajpurkar/squad`, 30,000 train examples |
| Precision | bf16 |
| Optimizer steps | 3,000 (batch 4 × grad-accum 8 = eff. batch 32) |
| LR / warmup | 3e-4, 300 warmup, weight decay 0.01 |
| Source / target max len | 384 / 32 |
| PEFT version | 0.19.1 |
| Seed | 37 |
| Hardware | 1× NVIDIA RTX 3060 (12 GB), local |

## Limitations

- English SQuAD-style extractive QA only; short answer spans grounded in the given context.
- Adapter requires `google-t5/t5-large` as the base model at load time.
- Evaluated on a held-out validation subset, not the official SQuAD test server.

## Citation

Part of an encoder–decoder vs decoder-only paradigm study (OptimiAI, 2026).