Eroux commited on
Commit
c3d1821
·
verified ·
1 Parent(s): dceae78

Update eval numbers to post-crop Hidden Trespass at validation-selected operating point

Browse files
Files changed (1) hide show
  1. README.md +28 -22
README.md CHANGED
@@ -25,8 +25,10 @@ footers, footnotes).
25
  - **Paper:** [buda-base/papers](https://github.com/buda-base/papers) (`papers/2026-tibetan-book-layout`) — *arXiv link forthcoming*
26
  - **Code:** [github.com/buda-base/tibla](https://github.com/buda-base/tibla)
27
 
28
- > This checkpoint is **seed 0**; across five training seeds the paper reports
29
- > mean F1 **0.961 ± 0.009** (this seed scores 0.959, just below the mean).
 
 
30
 
31
  ## Task
32
 
@@ -64,8 +66,8 @@ Footnote is deliberately kept low (recall-safe): the v4 test has only 38 footnot
64
  GT boxes, so a low threshold keeps recall near 1.0. Raising `header`/`footer` from
65
  0.25 to 0.60 lifts precision +0.028 for a −0.014 recall cost; raising `text-area`
66
  from 0.25 to 0.55 (native) lifts precision +0.007 at no recall cost. If you prefer
67
- one global knob, the single best-mean-F1 confidence is **0.74** (costs ≈0.008 mean
68
- F1 vs per-class tuning).
69
 
70
  ## Evaluation (TiBLAD v4, 833-page test)
71
 
@@ -73,36 +75,40 @@ F1 vs per-class tuning).
73
  |---|---|---|---|
74
  | license | **AGPL-3.0** | Apache-2.0 | Apache-2.0 |
75
  | base model | **RT-DETR-l (Ultralytics)** | PP-DocLayout-L (PaddleOCR, RT-DETR-L) | RF-DETR-L (Roboflow) |
76
- | mean F1 (canonical 3-class) | **0.959** | 0.958 | 0.927 |
77
- |   header-footer F1 | **0.952** | 0.951 | 0.949 |
78
- |   text-area F1 | **0.999** | 0.997 | 0.996 |
79
- |   footnote F1 | **0.925** | 0.925 | 0.835 |
80
  | mean AP@0.50 | **0.974** | 0.959 | 0.925 |
81
  | mean AP@[0.50:0.95] | **0.786** | 0.781 | 0.667 |
82
  | shared-class mAP@[.50:.95] (DocLayNet-aligned) | **0.650** | 0.641 | 0.604 |
83
- | Hidden Trespass — header/footer | **0.008** | 0.003 | 0.020 |
84
- | Hidden Trespass — footnote | **0.037** | 0.037 | 0.216 |
85
  | COTe (Trespass) | **0.975 (0.001)** | 0.978 (0.000) | 0.974 (0.002) |
86
- | operating confidence | **0.74** | 0.68 | 0.26 |
87
 
88
- *"operating confidence" is the single global best-mean-F1 confidence used for the
89
- reported F1.*
 
90
 
91
- **Hidden Trespass** = the missed peripheral (header/footer/footnote) ground-truth
92
- **area** that falls inside the predicted `text-area` crop; area-based,
93
- micro-averaged over the test set. Lower is better (less clutter bled into the OCR
94
- region). Formal definition in the [paper](https://github.com/buda-base/papers).
 
 
95
 
96
  ## Which checkpoint to pick
97
 
98
  | checkpoint | license | mean F1 | shared mAP | footnote HT |
99
  |---|---|---|---|---|
100
- | **TiBLA-RTDETR** (primary) | AGPL-3.0 | 0.959 | 0.650 | 0.037 |
101
- | TiBLA-PP-DocLayout-L | Apache-2.0 | 0.958 | 0.641 | 0.037 |
102
- | TiBLA-RFDETR | Apache-2.0 | 0.927 | 0.604 | 0.216 |
103
 
104
- RT-DETR-l has the top scores but its weights are AGPL-3.0 (Ultralytics). If you
105
- need a permissive license, PP-DocLayout-L matches it at Apache-2.0; RF-DETR is a
 
106
  lighter PyTorch-native Apache-2.0 option.
107
 
108
  ## Citation
 
25
  - **Paper:** [buda-base/papers](https://github.com/buda-base/papers) (`papers/2026-tibetan-book-layout`) — *arXiv link forthcoming*
26
  - **Code:** [github.com/buda-base/tibla](https://github.com/buda-base/tibla)
27
 
28
+ > This checkpoint is **seed 0**. Across five training seeds the paper reports
29
+ > mean canonical F1 **0.961 ± 0.009** (unified scorer, per-seed operating point);
30
+ > at the validation-selected operating point used in the table below this seed
31
+ > scores **0.952**.
32
 
33
  ## Task
34
 
 
66
  GT boxes, so a low threshold keeps recall near 1.0. Raising `header`/`footer` from
67
  0.25 to 0.60 lifts precision +0.028 for a −0.014 recall cost; raising `text-area`
68
  from 0.25 to 0.55 (native) lifts precision +0.007 at no recall cost. If you prefer
69
+ one global knob, the single validation-selected best-mean-F1 confidence is **0.64**
70
+ (costs ≈0.008 mean F1 vs per-class tuning).
71
 
72
  ## Evaluation (TiBLAD v4, 833-page test)
73
 
 
75
  |---|---|---|---|
76
  | license | **AGPL-3.0** | Apache-2.0 | Apache-2.0 |
77
  | base model | **RT-DETR-l (Ultralytics)** | PP-DocLayout-L (PaddleOCR, RT-DETR-L) | RF-DETR-L (Roboflow) |
78
+ | mean F1 (canonical 3-class) | **0.952** | 0.955 | 0.921 |
79
+ |   header-footer F1 | **0.954** | 0.953 | 0.947 |
80
+ |   text-area F1 | **0.999** | 0.998 | 0.996 |
81
+ |   footnote F1 | **0.902** | 0.914 | 0.821 |
82
  | mean AP@0.50 | **0.974** | 0.959 | 0.925 |
83
  | mean AP@[0.50:0.95] | **0.786** | 0.781 | 0.667 |
84
  | shared-class mAP@[.50:.95] (DocLayNet-aligned) | **0.650** | 0.641 | 0.604 |
85
+ | Hidden Trespass — header/footer | **0.009** | 0.004 | 0.021 |
86
+ | Hidden Trespass — footnote | **0.043** | 0.037 | 0.178 |
87
  | COTe (Trespass) | **0.975 (0.001)** | 0.978 (0.000) | 0.974 (0.002) |
88
+ | operating confidence | **0.64** | 0.61 | 0.47 |
89
 
90
+ *"operating confidence" is the single global best-mean-F1 confidence, selected on
91
+ the leak-free validation split and frozen for test (no test-set tuning). COCO AP
92
+ rows are threshold-free (all detections above the fixed 0.05 floor).*
93
 
94
+ **Hidden Trespass** = peripheral (header/footer/footnote) ground-truth **area**
95
+ that survives in the *actual OCR body crop* `C = E \ P`, where `E` is the predicted
96
+ `text-area` envelope and `P` is the union of the predicted peripheral boxes the
97
+ pipeline subtracts; area-based, micro-averaged over the test set. Lower is better
98
+ (less peripheral text bled into the OCR region). Formal definition in the
99
+ [paper](https://github.com/buda-base/papers).
100
 
101
  ## Which checkpoint to pick
102
 
103
  | checkpoint | license | mean F1 | shared mAP | footnote HT |
104
  |---|---|---|---|---|
105
+ | **TiBLA-RTDETR** (primary) | AGPL-3.0 | 0.952 | 0.650 | 0.043 |
106
+ | TiBLA-PP-DocLayout-L | Apache-2.0 | 0.955 | 0.641 | 0.037 |
107
+ | TiBLA-RFDETR | Apache-2.0 | 0.921 | 0.604 | 0.178 |
108
 
109
+ RT-DETR-l leads on mAP, shared-class mAP and the 5-seed mean F1 (0.961 ± 0.009),
110
+ but its weights are AGPL-3.0 (Ultralytics). If you need a permissive license,
111
+ PP-DocLayout-L is an Apache-2.0 match (statistically on par on F1); RF-DETR is a
112
  lighter PyTorch-native Apache-2.0 option.
113
 
114
  ## Citation