Fine-tune tiếp (max_input=4096, lr=5e-6, giữ #PersonN#)

README.md

@@ -24,16 +24,13 @@ model-index:
     metrics:
     - type: rouge
       name: ROUGE-1
-      value: 0.4632
+      value: 0.5036
     - type: rouge
       name: ROUGE-2
-      value: 0.2446
+      value: 0.3323
     - type: rouge
       name: ROUGE-L
-      value: 0.3970
-    - type: bertscore
-      name: BERTScore-F1
-      value: 0.8573
+      value: 0.4580
 ---
 
 # LongT5 — Meeting Summarization
@@ -76,30 +73,38 @@ hf download tuanhqv123/longt5-meeting-summarization --local-dir ./best_model
 
 ## Evaluation
 
-Held-out **test** set (342 examples):
+Held-out **test** set, current dataset revision (478 examples), beam=4, ROUGE with stemming, `#PersonN#` tags normalized in post-processing:
 
-| Metric       | Score  |
-|--------------|--------|
-| ROUGE-1      | 0.4632 |
-| ROUGE-2      | 0.2446 |
-| ROUGE-L      | 0.3970 |
-| BERTScore-F1 | 0.8573 |
-| Test loss    | 2.8342 |
+| Metric  | Score  |
+|---------|--------|
+| ROUGE-1 | 0.5036 |
+| ROUGE-2 | 0.3323 |
+| ROUGE-L | 0.4580 |
 
-This checkpoint replaces a previous production model and **improves on every metric**:
+### Per-source breakdown
 
-| Metric       | Previous | **This model** |
-|--------------|----------|----------------|
-| ROUGE-1      | 0.4645   | **0.4687**     |
-| ROUGE-2      | 0.2454   | **0.2500**     |
-| ROUGE-L      | 0.3991   | **0.4032**     |
-| BERTScore-F1 | 0.8567   | **0.8573**     |
+The two halves of the corpus behave very differently — reported separately for honesty:
 
-<sub>(comparison run on the same evaluation set with identical generation settings)</sub>
+| Source      | ROUGE-1 | ROUGE-2 | ROUGE-L |
+|-------------|---------|---------|---------|
+| MeetingBank | 0.6048  | 0.5135  | **0.5895** |
+| DialogSum   | 0.4017  | 0.1515  | 0.3264  |
+
+- **MeetingBank** drives the headline score (long council transcripts → short administrative summaries).
+- **DialogSum** plateaus around ROUGE-L ≈ 0.33 — and the base checkpoint, the merged fine-tune, and a DialogSum-only fine-tune all land within ±0.006 of each other, so this is a **capacity ceiling of `tglobal-base` on conversational reasoning**, not a data or training-recipe issue. See `REPORT.md` §9–10 in the training repo.
+
+### vs. base checkpoint (same 478-example test)
+
+| Metric  | base `./model` | **this fine-tune** |
+|---------|----------------|--------------------|
+| MeetingBank ROUGE-L | 0.528 | **0.590** |
+| DialogSum ROUGE-L   | 0.332 | 0.326 |
+
+Fine-tuning improves MeetingBank by ~0.06 ROUGE-L while leaving DialogSum essentially unchanged.
 
 ## Training
 
-**Data:** [`tuanhqv123/meeting-summarization`](https://huggingface.co/datasets/tuanhqv123/meeting-summarization) (DialogSum + MeetingBank).
+**Data:** [`tuanhqv123/meeting-summarization`](https://huggingface.co/datasets/tuanhqv123/meeting-summarization) — a 50/50 hybrid of **DialogSum** (short 2-person chats) and **MeetingBank** (long municipal-council transcripts). Current dataset card: **5,896 train / 594 val / 478 test** (6,968 total, balanced 3,484 / 3,484 per source).
 
 | Setting              | Value                          |
 |----------------------|--------------------------------|
@@ -112,10 +117,44 @@ This checkpoint replaces a previous production model and **improves on every met
 | Label smoothing      | 0.1                            |
 | Precision            | BF16                           |
 | Max input / output   | 16,384 / 512 tokens            |
-| Data split           | 3,757 train / 427 val / 342 test |
 | Hardware             | 1× NVIDIA RTX 4090             |
 | Training time        | ~3.7 h                         |
 
+### Data analysis (measured on the current dataset card)
+
+All numbers below are measured directly with this model's own tokenizer, not estimated.
+
+**The two sources have wildly different lengths** — they are not one homogeneous corpus:
+
+| Source | n | input tok (p50) | input tok (p99) | input tok (max) | output tok (p95) |
+|--------|---|-----------------|-----------------|-----------------|------------------|
+| DialogSum   | 3,484 | 204   | 600    | 1,494  | 73  |
+| MeetingBank | 3,484 | 1,304 | 33,742 | 84,354 | 201 |
+
+**Where does the summary's information live in the transcript?** (answer-coverage on 400 MeetingBank samples — the position, in tokens, of the latest summary content-word found in the input):
+
+| Truncate input at | % of summary content retained | % of meetings losing >20% |
+|-------------------|-------------------------------|---------------------------|
+| 2,048  | 95.8% | 7% |
+| **4,096**  | **97.9%** | **4%** |
+| 8,192  | 99.2% | 2% |
+| 16,384 | 99.8% | 0% |
+
+- Summary content concentrates early: the latest-referenced token sits at **p50 = 402, p75 = 1,853** — MeetingBank, like most minutes, front-loads its recommendation/decision.
+- **27.7%** of summary content-words never appear in the input at all (the targets are genuinely abstractive paraphrase, e.g. `"Recommendation to…"`), so 100% extractive coverage is impossible by construction.
+- **Conclusion:** `max_input = 4096` is the right operating point — going to 16,384 buys only +1.9% coverage for 4× the compute. Long samples are **truncated, not dropped**, so all 6,967 training examples are retained.
+
+### Preprocessing (deep clean)
+
+Pipeline keeps **100%** of usable rows (6,968 → 6,967; only 1 exact duplicate removed). Per-step before/after counts are reproduced by [`deep_preprocess.py`](deep_preprocess.py):
+
+- `html.unescape` + NFKC normalization, strip `{…}` annotations.
+- Normalize unicode quotes/dashes → ASCII (557 outputs touched).
+- **`#PersonN#` speaker tags are kept verbatim** to stay aligned with the original model's target format.
+- Remove disfluencies on the input side only (5,174 inputs touched — mostly MeetingBank fillers).
+- Filters: input ≥ 20 words, output 5–400 words, dedup on (input, output).
+- Non-English content characters (`ñ`, `é`, `•`, …) are preserved where they are real proper nouns (only 15 remain).
+
 ### Validation ROUGE per epoch
 
 | Epoch | ROUGE-1 | ROUGE-2 | ROUGE-L |

config.json

@@ -29,6 +29,6 @@
   "relative_attention_num_buckets": 32,
   "tie_word_embeddings": false,
   "transformers_version": "4.57.3",
-  "use_cache": true,
+  "use_cache": false,
   "vocab_size": 32128
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2860df3211f3fc6564d874e28fbc3d5122505bc6c093ba24064c0243e237d7e0
+oid sha256:e264bcbdeee99abea5a5fedba0ae6c42e0c06e52084fc5debe567ff59e17159c
 size 1187780840

special_tokens_map.json

@@ -101,7 +101,25 @@
     "<extra_id_98>",
     "<extra_id_99>"
   ],
-  "eos_token": "</s>",
-  "pad_token": "<pad>",
-  "unk_token": "<unk>"
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
 }

tokenizer.json

@@ -2,7 +2,7 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 512,
+    "max_length": 256,
     "strategy": "LongestFirst",
     "stride": 0
   },

tokenizer_config.json

@@ -932,8 +932,12 @@
   "eos_token": "</s>",
   "extra_ids": 100,
   "extra_special_tokens": {},
+  "max_length": 512,
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "<pad>",
+  "stride": 0,
   "tokenizer_class": "T5Tokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
   "unk_token": "<unk>"
 }