---
language: en
license: mit
tags:
- gpt2
- lora
- peft
- dialogue-summarization
- samsum
- unsloth
- lab-assignment
datasets:
- knkarthick/samsum
base_model: openai-community/gpt2
model-index:
- name: gpt2-lora-samsum
  results: []
---

# GPT‑2 LoRA – Dialogue Summarization (SAMSum)

This model is a **Parameter‑Efficient Fine‑Tuned (PEFT)** version of [GPT‑2](https://huggingface.co/openai-community/gpt2) using **Low‑Rank Adaptation (LoRA)**.  
It was trained on the [SAMSum dataset](https://huggingface.co/datasets/knkarthick/samsum) to generate concise summaries of messenger‑style dialogues.

## Intended Use

- **Primary task:** Dialogue summarisation  
- **Input:** A raw dialogue string (e.g., multi‑turn conversation)  
- **Output:** A short, fluent summary  
- **Example prompt format used during training:**  
  ```
  Summarize the following dialogue.

  Dialogue:
  {dialogue}

  Summary:
  ```

The model performs best on conversations similar to SAMSum (informal, chat‑like, English).

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "openai-community/gpt2"
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id)

model = PeftModel.from_pretrained(model, "violetar/gpt2-samsum-lora")

model = model.merge_and_unload()

def generate_summary(dialogue: str, max_new_tokens=128) -> str:
    prompt = f"Summarize the following dialogue.\n\nDialogue:\n{dialogue}\n\nSummary:\n"
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=768)
    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        do_sample=False,
        temperature=1.0,
        pad_token_id=tokenizer.eos_token_id,
    )
    summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
    # Remove prompt part
    summary = summary.split("Summary:\n")[-1].strip()
    return summary
```

## Training Details

- **Base model:** `openai-community/gpt2` (124M parameters)  
- **Dataset:** [SAMSum](https://huggingface.co/datasets/knkarthick/samsum) – 14,732 dialogues + summaries  
- **Training script:** Modified version of `gpt2.py` (provided in the lab)  
- **LoRA configuration:**  
  - `r = 32`  
  - `lora_alpha = 16`  
  - `lora_dropout = 0.0`  
  - Target modules: `["c_attn", "c_proj", "c_fc"]` (all GPT‑2 attention & FFN projections)  
- **Hyperparameters:**  
  - Max sequence length: 768 tokens  
  - Batch size: 4 (per device), gradient accumulation steps: 4 → effective batch size 16  
  - Learning rate: 2e‑4 (AdamW, warmup ratio 0.03)  
  - Epochs: 3  
  - Mixed precision: bf16 if supported, else fp16  
  - Optimizer: `adamw_torch`