Eroux commited on
Commit
74d2991
·
verified ·
1 Parent(s): fbce730

Update eval numbers to post-crop Hidden Trespass at validation-selected operating point

Browse files
Files changed (1) hide show
  1. README.md +27 -22
README.md CHANGED
@@ -44,15 +44,16 @@ operating point).
44
  from rfdetr import RFDETRLarge
45
 
46
  model = RFDETRLarge.from_checkpoint("rfdetr_tibetan_book_layout.pth")
47
- det = model.predict("page.jpg", threshold=0.26, shape=(1024, 1024))
48
  # checkpoint class ids are offset by 1 (id 0 = background):
49
  # 1 header, 2 text-area, 3 footnote, 4 footer
50
  ```
51
 
52
  A ready-made `infer.py` (batch, YOLO-format output, per-class thresholds) is
53
- included in this repo. Recommended global operating confidence: **0.26** (the
54
- best-mean-F1 point); the bundled `infer.py` also ships per-class max-F1 thresholds
55
- (`header` 0.46, `text-area` 0.32, `footnote` 0.26, `footer` 0.52).
 
56
 
57
  ## Evaluation (TiBLAD v4, 833-page test)
58
 
@@ -60,36 +61,40 @@ best-mean-F1 point); the bundled `infer.py` also ships per-class max-F1 threshol
60
  |---|---|---|---|
61
  | license | AGPL-3.0 | Apache-2.0 | **Apache-2.0** |
62
  | base model | RT-DETR-l (Ultralytics) | PP-DocLayout-L (PaddleOCR, RT-DETR-L) | **RF-DETR-L (Roboflow)** |
63
- | mean F1 (canonical 3-class) | 0.959 | 0.958 | **0.927** |
64
- |   header-footer F1 | 0.952 | 0.951 | **0.949** |
65
- |   text-area F1 | 0.999 | 0.997 | **0.996** |
66
- |   footnote F1 | 0.925 | 0.925 | **0.835** |
67
  | mean AP@0.50 | 0.974 | 0.959 | **0.925** |
68
  | mean AP@[0.50:0.95] | 0.786 | 0.781 | **0.667** |
69
  | shared-class mAP@[.50:.95] (DocLayNet-aligned) | 0.650 | 0.641 | **0.604** |
70
- | Hidden Trespass — header/footer | 0.008 | 0.003 | **0.020** |
71
- | Hidden Trespass — footnote | 0.037 | 0.037 | **0.216** |
72
  | COTe (Trespass) | 0.975 (0.001) | 0.978 (0.000) | **0.974 (0.002)** |
73
- | operating confidence | 0.74 | 0.68 | **0.26** |
74
 
75
- *"operating confidence" is the single global best-mean-F1 confidence used for the
76
- reported F1.*
 
77
 
78
- **Hidden Trespass** = the missed peripheral (header/footer/footnote) ground-truth
79
- **area** that falls inside the predicted `text-area` crop; area-based,
80
- micro-averaged over the test set. Lower is better (less clutter bled into the OCR
81
- region). Formal definition in the [paper](https://github.com/buda-base/papers).
 
 
82
 
83
  ## Which checkpoint to pick
84
 
85
  | checkpoint | license | mean F1 | shared mAP | footnote HT |
86
  |---|---|---|---|---|
87
- | TiBLA-RTDETR (primary) | AGPL-3.0 | 0.959 | 0.650 | 0.037 |
88
- | TiBLA-PP-DocLayout-L | Apache-2.0 | 0.958 | 0.641 | 0.037 |
89
- | **TiBLA-RFDETR** | Apache-2.0 | 0.927 | 0.604 | 0.216 |
90
 
91
- RT-DETR-l has the top scores but its weights are AGPL-3.0 (Ultralytics). If you
92
- need a permissive license, PP-DocLayout-L matches it at Apache-2.0; RF-DETR is a
 
93
  lighter PyTorch-native Apache-2.0 option.
94
 
95
  ## Citation
 
44
  from rfdetr import RFDETRLarge
45
 
46
  model = RFDETRLarge.from_checkpoint("rfdetr_tibetan_book_layout.pth")
47
+ det = model.predict("page.jpg", threshold=0.47, shape=(1024, 1024))
48
  # checkpoint class ids are offset by 1 (id 0 = background):
49
  # 1 header, 2 text-area, 3 footnote, 4 footer
50
  ```
51
 
52
  A ready-made `infer.py` (batch, YOLO-format output, per-class thresholds) is
53
+ included in this repo. Recommended global operating confidence: **0.47** (the
54
+ validation-selected best-mean-F1 point); the bundled `infer.py` also ships
55
+ per-class max-F1 thresholds (`header` 0.46, `text-area` 0.32, `footnote` 0.26,
56
+ `footer` 0.52).
57
 
58
  ## Evaluation (TiBLAD v4, 833-page test)
59
 
 
61
  |---|---|---|---|
62
  | license | AGPL-3.0 | Apache-2.0 | **Apache-2.0** |
63
  | base model | RT-DETR-l (Ultralytics) | PP-DocLayout-L (PaddleOCR, RT-DETR-L) | **RF-DETR-L (Roboflow)** |
64
+ | mean F1 (canonical 3-class) | 0.952 | 0.955 | **0.921** |
65
+ |   header-footer F1 | 0.954 | 0.953 | **0.947** |
66
+ |   text-area F1 | 0.999 | 0.998 | **0.996** |
67
+ |   footnote F1 | 0.902 | 0.914 | **0.821** |
68
  | mean AP@0.50 | 0.974 | 0.959 | **0.925** |
69
  | mean AP@[0.50:0.95] | 0.786 | 0.781 | **0.667** |
70
  | shared-class mAP@[.50:.95] (DocLayNet-aligned) | 0.650 | 0.641 | **0.604** |
71
+ | Hidden Trespass — header/footer | 0.009 | 0.004 | **0.021** |
72
+ | Hidden Trespass — footnote | 0.043 | 0.037 | **0.178** |
73
  | COTe (Trespass) | 0.975 (0.001) | 0.978 (0.000) | **0.974 (0.002)** |
74
+ | operating confidence | 0.64 | 0.61 | **0.47** |
75
 
76
+ *"operating confidence" is the single global best-mean-F1 confidence, selected on
77
+ the leak-free validation split and frozen for test (no test-set tuning). COCO AP
78
+ rows are threshold-free (all detections above the fixed 0.05 floor).*
79
 
80
+ **Hidden Trespass** = peripheral (header/footer/footnote) ground-truth **area**
81
+ that survives in the *actual OCR body crop* `C = E \ P`, where `E` is the predicted
82
+ `text-area` envelope and `P` is the union of the predicted peripheral boxes the
83
+ pipeline subtracts; area-based, micro-averaged over the test set. Lower is better
84
+ (less peripheral text bled into the OCR region). Formal definition in the
85
+ [paper](https://github.com/buda-base/papers).
86
 
87
  ## Which checkpoint to pick
88
 
89
  | checkpoint | license | mean F1 | shared mAP | footnote HT |
90
  |---|---|---|---|---|
91
+ | TiBLA-RTDETR (primary) | AGPL-3.0 | 0.952 | 0.650 | 0.043 |
92
+ | TiBLA-PP-DocLayout-L | Apache-2.0 | 0.955 | 0.641 | 0.037 |
93
+ | **TiBLA-RFDETR** | Apache-2.0 | 0.921 | 0.604 | 0.178 |
94
 
95
+ RT-DETR-l leads on mAP, shared-class mAP and the 5-seed mean F1 (0.961 ± 0.009),
96
+ but its weights are AGPL-3.0 (Ultralytics). If you need a permissive license,
97
+ PP-DocLayout-L is an Apache-2.0 match (statistically on par on F1); RF-DETR is a
98
  lighter PyTorch-native Apache-2.0 option.
99
 
100
  ## Citation