add BERTScore-F1 to evaluation

b6d8c23 verified 3 months ago

5.13 kB

license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: summarization
base_model: google/long-t5-tglobal-base
datasets:
  - knkarthick/dialogsum
  - knkarthick/samsum
tags:
  - longt5
  - summarization
  - dialogue-summarization
metrics:
  - rouge
model-index:
  - name: longt5-dialogue-summarization
    results:
      - task:
          type: summarization
          name: Dialogue Summarization
        metrics:
          - type: rouge
            name: ROUGE-1
            value: 0.4738
          - type: rouge
            name: ROUGE-2
            value: 0.2316
          - type: rouge
            name: ROUGE-L
            value: 0.3913
          - type: bertscore
            name: BERTScore-F1
            value: 0.9125

LongT5 — Dialogue Summarization

A LongT5 (tglobal-base) model fine-tuned to summarize short conversations into a sentence or two. Trained on DialogSum + SAMSum — two-person chats and group messenger threads.

Base model: google/long-t5-tglobal-base
Task: Abstractive dialogue summarization (English)
Max input / output: 512 / 96 tokens
License: Apache-2.0

🔧 Reproducibility: the pipeline lives in code/ as two runnable notebooks — data_processing.ipynb (clean + EDA + investigation) and train.ipynb (fine-tune + evaluate) — plus code/REPORT.md for the full analysis and decision log.

Quick start

The model was trained with a "summarize: " task prefix — add it at inference too:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

repo = "tuanhqv123/longt5-meeting-summarization"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo)

dialogue = """Person1: Can I help you?
Person2: I'd like to buy a new mobile phone please."""
inputs = tok("summarize: " + dialogue, return_tensors="pt", truncation=True, max_length=512)
ids = model.generate(**inputs, max_new_tokens=96, num_beams=4, no_repeat_ngram_size=3)
print(tok.decode(ids[0], skip_special_tokens=True))

Evaluation

Held-out test set (DialogSum + SAMSum), beam=4, ROUGE with stemming + BERTScore-F1:

Metric	Score
ROUGE-1	0.4738
ROUGE-2	0.2316
ROUGE-L	0.3913
BERTScore-F1	0.9125

ROUGE measures word overlap; BERTScore measures semantic similarity, so its higher value reflects that the summaries are usually correct in meaning even when worded differently.

Per-source breakdown

Source	ROUGE-1	ROUGE-2	ROUGE-L	BERTScore-F1
SAMSum	0.5026	0.2650	0.4231	0.9156
DialogSum	0.4521	0.1873	0.3645	0.9100

SAMSum (casual messenger chats) summarizes a bit more cleanly than DialogSum (longer, more structured two-person dialogues).

Training

Data: knkarthick/dialogsum + knkarthick/samsum, cleaned and merged (29,610 rows after cleaning). DialogSum's #Person1# tags are normalized to Person1 so the two sources share a consistent speaker style. Each dataset's original train/val/test split is kept.

Setting	Value
Base model	`google/long-t5-tglobal-base`
Task prefix	`summarize:`
Epochs	max 10, early stopping (patience 2) → best at epoch 6
Batch size	16
Learning rate	3e-4
Warmup steps	200
Label smoothing	0.1
Precision	BF16
Max input / output	512 / 96 tokens
Hardware	1× NVIDIA RTX 4090

Validation ROUGE-L per epoch

Early stopping (patience 2) picked epoch 6 — validation ROUGE-L peaked there, then didn't improve.

Epoch	1	2	3	4	5	6	7	8
ROUGE-L	0.4119	0.4261	0.4275	0.4258	0.4335	0.4349	0.4316	0.4322

Intended use & limitations

Intended: abstractive summarization of short English conversations / chat threads.
Limitations: English-only; trained on casual/everyday dialogue, so it may transfer poorly to technical, legal, or very long transcripts. Like all abstractive summarizers it can hallucinate — verify facts before relying on a summary. Inputs beyond 512 tokens are truncated.

Citation

Built on LongT5:

@article{guo2021longt5,
  title={LongT5: Efficient Text-To-Text Transformer for Long Sequences},
  author={Guo, Mandy and Ainslie, Joshua and Uthus, David and Ontanon, Santiago and Ni, Jianmo and Sung, Yun-Hsuan and Yang, Yinfei},
  journal={arXiv preprint arXiv:2112.07916},
  year={2021}
}