add BERTScore-F1 to evaluation
Browse files
README.md
CHANGED
|
@@ -30,6 +30,9 @@ model-index:
|
|
| 30 |
- type: rouge
|
| 31 |
name: ROUGE-L
|
| 32 |
value: 0.3913
|
|
|
|
|
|
|
|
|
|
| 33 |
---
|
| 34 |
|
| 35 |
# LongT5 — Dialogue Summarization
|
|
@@ -66,20 +69,24 @@ print(tok.decode(ids[0], skip_special_tokens=True))
|
|
| 66 |
|
| 67 |
## Evaluation
|
| 68 |
|
| 69 |
-
Held-out test set (DialogSum + SAMSum), beam=4, ROUGE with stemming:
|
| 70 |
|
| 71 |
-
| Metric
|
| 72 |
-
|---------|--------|
|
| 73 |
-
| ROUGE-1
|
| 74 |
-
| ROUGE-2
|
| 75 |
-
| ROUGE-L
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
### Per-source breakdown
|
| 78 |
|
| 79 |
-
| Source | ROUGE-1 | ROUGE-2 | ROUGE-L |
|
| 80 |
-
|-----------|---------|---------|---------|
|
| 81 |
-
| SAMSum | 0.5026 | 0.2650 | **0.4231** |
|
| 82 |
-
| DialogSum | 0.4521 | 0.1873 | 0.3645 |
|
| 83 |
|
| 84 |
SAMSum (casual messenger chats) summarizes a bit more cleanly than DialogSum (longer, more
|
| 85 |
structured two-person dialogues).
|
|
|
|
| 30 |
- type: rouge
|
| 31 |
name: ROUGE-L
|
| 32 |
value: 0.3913
|
| 33 |
+
- type: bertscore
|
| 34 |
+
name: BERTScore-F1
|
| 35 |
+
value: 0.9125
|
| 36 |
---
|
| 37 |
|
| 38 |
# LongT5 — Dialogue Summarization
|
|
|
|
| 69 |
|
| 70 |
## Evaluation
|
| 71 |
|
| 72 |
+
Held-out test set (DialogSum + SAMSum), beam=4, ROUGE with stemming + BERTScore-F1:
|
| 73 |
|
| 74 |
+
| Metric | Score |
|
| 75 |
+
|--------------|--------|
|
| 76 |
+
| ROUGE-1 | 0.4738 |
|
| 77 |
+
| ROUGE-2 | 0.2316 |
|
| 78 |
+
| ROUGE-L | 0.3913 |
|
| 79 |
+
| BERTScore-F1 | 0.9125 |
|
| 80 |
+
|
| 81 |
+
ROUGE measures word overlap; BERTScore measures semantic similarity, so its higher value reflects
|
| 82 |
+
that the summaries are usually correct in meaning even when worded differently.
|
| 83 |
|
| 84 |
### Per-source breakdown
|
| 85 |
|
| 86 |
+
| Source | ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore-F1 |
|
| 87 |
+
|-----------|---------|---------|---------|--------------|
|
| 88 |
+
| SAMSum | 0.5026 | 0.2650 | **0.4231** | **0.9156** |
|
| 89 |
+
| DialogSum | 0.4521 | 0.1873 | 0.3645 | 0.9100 |
|
| 90 |
|
| 91 |
SAMSum (casual messenger chats) summarizes a bit more cleanly than DialogSum (longer, more
|
| 92 |
structured two-person dialogues).
|