Update eval numbers to post-crop Hidden Trespass at validation-selected operating point
Browse files
README.md
CHANGED
|
@@ -44,15 +44,16 @@ operating point).
|
|
| 44 |
from rfdetr import RFDETRLarge
|
| 45 |
|
| 46 |
model = RFDETRLarge.from_checkpoint("rfdetr_tibetan_book_layout.pth")
|
| 47 |
-
det = model.predict("page.jpg", threshold=0.
|
| 48 |
# checkpoint class ids are offset by 1 (id 0 = background):
|
| 49 |
# 1 header, 2 text-area, 3 footnote, 4 footer
|
| 50 |
```
|
| 51 |
|
| 52 |
A ready-made `infer.py` (batch, YOLO-format output, per-class thresholds) is
|
| 53 |
-
included in this repo. Recommended global operating confidence: **0.
|
| 54 |
-
best-mean-F1 point); the bundled `infer.py` also ships
|
| 55 |
-
(`header` 0.46, `text-area` 0.32, `footnote` 0.26,
|
|
|
|
| 56 |
|
| 57 |
## Evaluation (TiBLAD v4, 833-page test)
|
| 58 |
|
|
@@ -60,36 +61,40 @@ best-mean-F1 point); the bundled `infer.py` also ships per-class max-F1 threshol
|
|
| 60 |
|---|---|---|---|
|
| 61 |
| license | AGPL-3.0 | Apache-2.0 | **Apache-2.0** |
|
| 62 |
| base model | RT-DETR-l (Ultralytics) | PP-DocLayout-L (PaddleOCR, RT-DETR-L) | **RF-DETR-L (Roboflow)** |
|
| 63 |
-
| mean F1 (canonical 3-class) | 0.
|
| 64 |
-
| header-footer F1 | 0.
|
| 65 |
-
| text-area F1 | 0.999 | 0.
|
| 66 |
-
| footnote F1 | 0.
|
| 67 |
| mean AP@0.50 | 0.974 | 0.959 | **0.925** |
|
| 68 |
| mean AP@[0.50:0.95] | 0.786 | 0.781 | **0.667** |
|
| 69 |
| shared-class mAP@[.50:.95] (DocLayNet-aligned) | 0.650 | 0.641 | **0.604** |
|
| 70 |
-
| Hidden Trespass — header/footer | 0.
|
| 71 |
-
| Hidden Trespass — footnote | 0.
|
| 72 |
| COTe (Trespass) | 0.975 (0.001) | 0.978 (0.000) | **0.974 (0.002)** |
|
| 73 |
-
| operating confidence | 0.
|
| 74 |
|
| 75 |
-
*"operating confidence" is the single global best-mean-F1 confidence
|
| 76 |
-
|
|
|
|
| 77 |
|
| 78 |
-
**Hidden Trespass** =
|
| 79 |
-
**
|
| 80 |
-
|
| 81 |
-
|
|
|
|
|
|
|
| 82 |
|
| 83 |
## Which checkpoint to pick
|
| 84 |
|
| 85 |
| checkpoint | license | mean F1 | shared mAP | footnote HT |
|
| 86 |
|---|---|---|---|---|
|
| 87 |
-
| TiBLA-RTDETR (primary) | AGPL-3.0 | 0.
|
| 88 |
-
| TiBLA-PP-DocLayout-L | Apache-2.0 | 0.
|
| 89 |
-
| **TiBLA-RFDETR** | Apache-2.0 | 0.
|
| 90 |
|
| 91 |
-
RT-DETR-l
|
| 92 |
-
|
|
|
|
| 93 |
lighter PyTorch-native Apache-2.0 option.
|
| 94 |
|
| 95 |
## Citation
|
|
|
|
| 44 |
from rfdetr import RFDETRLarge
|
| 45 |
|
| 46 |
model = RFDETRLarge.from_checkpoint("rfdetr_tibetan_book_layout.pth")
|
| 47 |
+
det = model.predict("page.jpg", threshold=0.47, shape=(1024, 1024))
|
| 48 |
# checkpoint class ids are offset by 1 (id 0 = background):
|
| 49 |
# 1 header, 2 text-area, 3 footnote, 4 footer
|
| 50 |
```
|
| 51 |
|
| 52 |
A ready-made `infer.py` (batch, YOLO-format output, per-class thresholds) is
|
| 53 |
+
included in this repo. Recommended global operating confidence: **0.47** (the
|
| 54 |
+
validation-selected best-mean-F1 point); the bundled `infer.py` also ships
|
| 55 |
+
per-class max-F1 thresholds (`header` 0.46, `text-area` 0.32, `footnote` 0.26,
|
| 56 |
+
`footer` 0.52).
|
| 57 |
|
| 58 |
## Evaluation (TiBLAD v4, 833-page test)
|
| 59 |
|
|
|
|
| 61 |
|---|---|---|---|
|
| 62 |
| license | AGPL-3.0 | Apache-2.0 | **Apache-2.0** |
|
| 63 |
| base model | RT-DETR-l (Ultralytics) | PP-DocLayout-L (PaddleOCR, RT-DETR-L) | **RF-DETR-L (Roboflow)** |
|
| 64 |
+
| mean F1 (canonical 3-class) | 0.952 | 0.955 | **0.921** |
|
| 65 |
+
| header-footer F1 | 0.954 | 0.953 | **0.947** |
|
| 66 |
+
| text-area F1 | 0.999 | 0.998 | **0.996** |
|
| 67 |
+
| footnote F1 | 0.902 | 0.914 | **0.821** |
|
| 68 |
| mean AP@0.50 | 0.974 | 0.959 | **0.925** |
|
| 69 |
| mean AP@[0.50:0.95] | 0.786 | 0.781 | **0.667** |
|
| 70 |
| shared-class mAP@[.50:.95] (DocLayNet-aligned) | 0.650 | 0.641 | **0.604** |
|
| 71 |
+
| Hidden Trespass — header/footer | 0.009 | 0.004 | **0.021** |
|
| 72 |
+
| Hidden Trespass — footnote | 0.043 | 0.037 | **0.178** |
|
| 73 |
| COTe (Trespass) | 0.975 (0.001) | 0.978 (0.000) | **0.974 (0.002)** |
|
| 74 |
+
| operating confidence | 0.64 | 0.61 | **0.47** |
|
| 75 |
|
| 76 |
+
*"operating confidence" is the single global best-mean-F1 confidence, selected on
|
| 77 |
+
the leak-free validation split and frozen for test (no test-set tuning). COCO AP
|
| 78 |
+
rows are threshold-free (all detections above the fixed 0.05 floor).*
|
| 79 |
|
| 80 |
+
**Hidden Trespass** = peripheral (header/footer/footnote) ground-truth **area**
|
| 81 |
+
that survives in the *actual OCR body crop* `C = E \ P`, where `E` is the predicted
|
| 82 |
+
`text-area` envelope and `P` is the union of the predicted peripheral boxes the
|
| 83 |
+
pipeline subtracts; area-based, micro-averaged over the test set. Lower is better
|
| 84 |
+
(less peripheral text bled into the OCR region). Formal definition in the
|
| 85 |
+
[paper](https://github.com/buda-base/papers).
|
| 86 |
|
| 87 |
## Which checkpoint to pick
|
| 88 |
|
| 89 |
| checkpoint | license | mean F1 | shared mAP | footnote HT |
|
| 90 |
|---|---|---|---|---|
|
| 91 |
+
| TiBLA-RTDETR (primary) | AGPL-3.0 | 0.952 | 0.650 | 0.043 |
|
| 92 |
+
| TiBLA-PP-DocLayout-L | Apache-2.0 | 0.955 | 0.641 | 0.037 |
|
| 93 |
+
| **TiBLA-RFDETR** | Apache-2.0 | 0.921 | 0.604 | 0.178 |
|
| 94 |
|
| 95 |
+
RT-DETR-l leads on mAP, shared-class mAP and the 5-seed mean F1 (0.961 ± 0.009),
|
| 96 |
+
but its weights are AGPL-3.0 (Ultralytics). If you need a permissive license,
|
| 97 |
+
PP-DocLayout-L is an Apache-2.0 match (statistically on par on F1); RF-DETR is a
|
| 98 |
lighter PyTorch-native Apache-2.0 option.
|
| 99 |
|
| 100 |
## Citation
|