medelharchaoui/t5-large-lora-r8-squad

T5-large — LoRA r=8 adapter on SQuAD

A LoRA adapter (PEFT) for google-t5/t5-large, trained for extractive QA on SQuAD. Rank 8 on the q and v projections only → ~2.4M trainable params (0.32% of the model).

TL;DR: this 9.5 MB adapter reaches F1 0.8152 / EM 0.6445 — statistically tied with full fine-tuning (F1 0.8162) and with the cross-attention-only variant (F1 0.8128), and it beats parameter-matched GPT-2-large full fine-tune (F1 0.5041) by +0.31 F1 while training 0.32% of the weights versus its 100%. Architecture alignment, not the number of tuned parameters, drives the result.

This repo contains the PEFT adapter only (adapter_model.safetensors, ~9.5 MB). The base weights are pulled from google-t5/t5-large at load time.

Results (SQuAD validation)

Model	Architecture	Trainable / Total	EM	Token F1
GPT-2-large	decoder-only	774M / 774M	0.3516	0.5041
T5-large XA-only	enc-dec	100.7M / 737M	0.6406	0.8128
T5-large LoRA r=8 (this adapter)	enc-dec	2.4M / 740M	0.6445	0.8152
T5-large full fine-tune	enc-dec	737M / 737M	0.6602	0.8162

Validation loss 0.3061, perplexity 1.36.

Eval note: T5 numbers use a 256-example SQuAD-validation generation subset (4-beam search); GPT-2-large uses 512. Magnitudes are comparable and reproduce the paper's ordering.

How to use

Trained with the input prefix answer question: prepended to a question: ... context: ... source string — match it exactly at inference:

from transformers import T5ForConditionalGeneration, AutoTokenizer
from peft import PeftModel

base = "google-t5/t5-large"
adapter = "medelharchaoui/t5-large-lora-r8-squad"

tok = AutoTokenizer.from_pretrained(adapter)
model = T5ForConditionalGeneration.from_pretrained(base)
model = PeftModel.from_pretrained(model, adapter)
model = model.merge_and_unload()  # optional: fold LoRA into base for faster inference

question = "What culture do 'bairn' and 'hyem' originate from?"
context = ("'bairn' and 'hyem' are geordie words with origins in scandinavia; barn and hjem "
           "are the corresponding modern norwegian and danish words.")
text = f"answer question: question: {question} context: {context}"
ids = tok(text, return_tensors="pt", truncation=True, max_length=384).input_ids
print(tok.decode(model.generate(ids, num_beams=4, max_new_tokens=16)[0], skip_special_tokens=True))

Training

Setting	Value
Base model	google-t5/t5-large (737M)
LoRA	r=8, alpha=32, dropout=0.05, target modules ["q", "v"]
Trainable params	~2.4M (0.32%)
Dataset	rajpurkar/squad, 30,000 train examples
Precision	bf16
Optimizer steps	3,000 (batch 4 × grad-accum 8 = eff. batch 32)
LR / warmup	3e-4, 300 warmup, weight decay 0.01
Source / target max len	384 / 32
PEFT version	0.19.1
Seed	37
Hardware	1× NVIDIA RTX 3060 (12 GB), local

Limitations

• English SQuAD-style extractive QA only; short answer spans grounded in the given context.
• Adapter requires google-t5/t5-large as the base model at load time.
• Evaluated on a held-out validation subset, not the official SQuAD test server.

Citation

Part of an encoder–decoder vs decoder-only paradigm study (OptimiAI, 2026).