Known limitations — real failure cases
The model is recall-tuned and OCR-robust, but not perfect. Below are honest, reproducible failure cases so you know where to add safeguards.
- All inputs are synthetic — fictional names, no real personal data and no evaluation/validation data is shared here.
- Outputs are the model's actual predictions (quantized ONNX, recall-first PER
threshold 0.2). Reproduce with
known_limitations.py. - The leak that matters is identity-level: if any mention of a person is missed (or only partially caught), that person can be re-identified.
1. Heavy OCR garble → name missed entirely
When OCR destroys most of a name's recognizable structure, the model can fail to flag it at all. This is the model's main remaining weakness.
| Input | Behind the garble | Model PER |
|---|---|---|
Pozwana ote tobodaaka wniosła sprzeciw. |
Bożena Łobodzińska | — (nothing) |
Powód go Hea stawił się osobiście. |
Igor Heliasz | — (nothing) |
Decyzję doręczono: ist yi daziak. |
Justyna Idziak | — (nothing) |
2. Partial detection → residue leak
The model catches part of a garbled name but not all of it, so a fragment of the real name stays visible after masking.
| Input | Behind the garble | Model PER |
Problem |
|---|---|---|---|
Wniosek złożył wiar Żoją w dniu 5 maja. |
Wiktor Żołądek | ['Żoją'] |
first name wiar left unmasked |
WIESEAW KUE stawił się na rozprawie. |
Wiesław Kuc | ['WIE', 'AW KUE'] |
split into two spans → the gap (SE) leaks |
Other known behaviours
- Foreign / out-of-distribution names (e.g. German, Dutch) are less reliable — the model is tuned for Polish.
- Boundaries can occasionally over- or under-extend (e.g. an adjacent honorific or a stray punctuation token captured with the name).
- On clean digital text this OCR-robust variant is slightly less precise than a non-OCR-augmented model (it is more eager on garbled-looking tokens).
Mitigations (recommended)
- Document-safety post-pass — once any clean mention of a surname is detected in a document, propagate masking to its other inflected / OCR-variant mentions (fuzzy match). This recovers many of the partial/residue cases above when a cleaner mention exists elsewhere in the same document.
- Checksum regex for structured PII (PESEL, NIP, IBAN, …) — deterministic and not subject to these OCR misses.
- Lower the threshold on noisy/scanned documents, or route them through OCR quality checks.
- Human review for high-stakes anonymisation — the model is a strong first pass, not a guarantee.