README.md

---
license: mit
language:
- pl
base_model: FacebookAI/xlm-roberta-large
library_name: transformers
pipeline_tag: token-classification
tags:
- polish
- pii
- ner
- xlm-roberta
- hybrid
---

# NERGAL 1.1.2

**Named Entity Recognition with Grounded Additive Labels**

SlayerLab hybrid PII cleaner for Polish. Not a chat model. Not a drop-in `pipeline("token-classification")`.

## TL;DR

Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`.

- **Version:** `1.1.2` (`hybrid.json`, `CHANGELOG.md`)
- **Ground:** `scrub_pii` regex (SHA256 `08faef84…`)
- **Additive labels:** XLM-RoBERTa-large token classifier, BIO tags `phone` / `pii`, threshold 0.95
- **Throughput:** about 80k chars/s on one RTX 4090 with `scrub_many` + `dtype="float16"` and 3 processes (1.0.3: 23k)
- **This snapshot:** seed `202609160`, **epoch 5** of a seven-epoch schedule

## What NERGAL detects — and what it does not

NERGAL masks **contact details and selected identifiers in Polish text**. It is not a general-purpose anonymizer: names, postal addresses and other personal information can remain in the output. Its corpus-masking policy also includes public, institutional and company contacts and identifiers.

### Detection scope

These are target categories, not a guarantee that every occurrence or format is detected.

| Category | Values in scope | Replacement |
|---|---|---|
| Phone contacts | Phone, fax and SMS contact numbers, including foreign, emergency, short/service and vanity numbers; extensions and number alternatives | `[Telefon]` |
| Email | Email addresses, including recoverable broken or incomplete addresses | `[PII]` |
| Personal identifiers | PESEL, passport and identity-document numbers | `[PII]` |
| Organization identifiers | NIP/VAT, REGON, KRS, GEMI, LEI and equivalent foreign company registration numbers; labelled DUNS, BDO and RPWDL register-book numbers | `[PII]` |
| Financial identifiers | Bank/account numbers, including Polish accounts and foreign IBANs | `[PII]` |
| Property identifiers | Land-register (księga wieczysta, KW) numbers | `[PII]` |
| Electronic contacts and access | e-Doręczenia and ePUAP addresses, explicitly labelled numeric access PINs (including `pin=` URL values), GG account IDs | `[PII]` |

Context matters: a number that resembles a phone or identifier is not automatically in scope. Rules use labels, format checks and, for some unlabelled identifiers, checksums; the model adds contextual detections. Coverage varies by category, and the aggregate benchmark below does not establish recall for every category.

### Outside detection scope

NERGAL is not designed to remove:

- **Personal names**, including private individuals and public officials; organization names.
- **Postal/street addresses, dates of birth and ages.**
- **Social-media handles, ordinary URLs and filenames.** An in-scope value inside a URL, such as a labelled numeric PIN, can still be masked.
- **Vehicle registration plates and generic serial, model or version codes.**
- **Document, case, article, funding and procurement references**, including procedure UUIDs; ISBNs, ORCIDs, TERYT codes, EAN/GTIN product codes and CNIL website-registration references.
- Prices, list numbers, generic labels without values, clearly fictitious examples and already-redacted placeholders.

These are intended exclusions; false positives can still mask some of this content.

### Known gaps in 1.1.2

Unlabelled phones and identifiers (the rules take a bare phone only in the grouped national forms `601 234 567` and `22 123 45 67`; a plain `601234567` needs a label or the model), unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 1.1.2 model. Do not rely on it to remove them consistently. An email with a lowercase word glued onto its domain (`jan@firma.plkontakt`) is masked together with that word, and text glued before an address can be masked with it.

## Versions

841-dev, union at 0.95, 354 gold spans. Same weights and API is a patch; new capability is minor; API, threshold, or weight recipe is major. A release that changes these numbers updates `hybrid.json` `eval` and this table. From 1.1.2 on, one 841-dev span is amended by a reviewed gold check (`gold_check_v2`, 10 characters trimmed); 1.1.1 is restated on it, and earlier rows use the original gold.

| Version | Whole /354 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
|---|---:|---:|---:|---:|---:|---:|---|
| 1.0.0 | 323 | 25 | 133 | 133 | 97.76% | 95.95% | First Hub snapshot |
| 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
| 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
| 1.0.3 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Label-note, e-Delivery and registry rules; placeholder and card fixes |
| 1.1.0 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Batch API (`predict_many`, `scrub_many`), opt-in float16 |
| 1.1.1 | 326 | 23 | 98 | 123 | 97.94% | 96.50% | Grouped national phones without a label |
| 1.1.1, amended gold | 326 | 23 | 108 | 133 | 97.77% | 96.49% | Restated on `gold_check_v2` |
| **1.1.2** | **326** | **23** | **24** | **80** | **98.65%** | **96.49%** | Email boundary rules |

## Cue-less phone test

841-dev holds few phones without a label, so it barely moves for 1.1.1. This targeted test does: 150 web passages selected for phones with no contact label, labelled and frozen before any prediction, and scored once (2026-09-29). Union at 0.95, same model spans for both.

| | 1.1.0 | **1.1.1** |
|---|---:|---:|
| Phones wholly masked /111 | 37 | **66** |
| All values wholly masked /178 | 98 | **127** |
| Positive passages fully covered /70 (95% CI) | 24 (0.23–0.47) | **37 (0.41–0.65)** |
| Passages with false masks /150 | 3 | 4 |
| False characters | 83 | 95 |

1.1.1 lost no value 1.1.0 masked; its 12 new false characters are in one passage. The set is enriched by selection, so these numbers say nothing about how common such phones are, and it has a single reviewer. 1.1.2 changes one passage (an email match): false characters 95 → 78, phones unchanged.

## 841-dev

Tables use one development split: 841 passages, 215 with gold PII, **354 spans** (169 phone, 185 other PII). It is the `dev` side of a 4,500-passage labelled tranche (3,655 train / 841 dev). Sources match Dynaword (EUR-Lex, HPLT, Wikipedia, parliamentary and government text, plus smaller news/literary slices). Labels mix unchanged silver with human review.

One span, an address that ran on into a glued URL host, was trimmed by a reviewed post-hoc gold check (`gold_check_v2`); numbers from 1.1.2 on use the amended gold.

The files contain real identifiers, so they are not released with the weights.

## Why XLM-R

GLiNER, HerBERT-large, and XLM-R-large were trained on the same split and unioned with the same regex. Plot: diagnostic threshold 0.50; selection used the full threshold grid. GLiNER covers more at 0.50 and then dumps precision. XLM-R is the architecture we kept.

![Primary three-model curves](figures/primary-three-model-curves.png)

## Why epoch 5

Fresh XLM-R, seven epochs. 133 epoch/threshold combinations. Epoch 5 at 0.95 was the only point that both beat the historical GLiNER∪regex incumbent on coverage and introduced zero new false-mask characters. Epoch 7 covers more PII (334/354) but adds 15 new false characters.

![Seven-epoch XLM-R curves](figures/xlmr-seven-epoch-curves.png)

| Epoch | Covered @ 0.95 /354 | Residual passages | False characters | New false vs incumbent |
|---|---:|---:|---:|---:|
| 1 | 272 | 59 | 175 | 42 |
| 2 | 291 | 49 | 141 | 8 |
| 3 | 317 | 30 | 140 | 7 |
| 4 | 320 | 28 | 143 | 10 |
| **5** | **323** | **25** | **133** | **0** |
| 6 | 328 | 20 | 147 | 14 |
| 7 | 334 | 16 | 148 | 15 |

## Compared with other systems

Same 841-dev split, threshold **0.95**. **Naked** is the transformer alone. **∪ regex** is that model unioned with the current rules. Character scores are gold vs masked characters.

| System | Mode | Whole /354 | Residual | False chars | Char P | Char R |
|---|---|---:|---:|---:|---:|---:|
| Regex (`scrub_pii`) | rules | 263 | 68 | 24 | 99.54% | 85.76% |
| GLiNER 2.5-multi zero-shot | naked | 81 | 188 | 970 | 56.98% | 21.22% |
| GLiNER 2.5-multi zero-shot | ∪ regex | 273 | 61 | 1,068 | 83.33% | 88.16% |
| Historical GLiNER email12 | naked | 249 | 70 | 10 | 99.81% | 85.39% |
| Historical GLiNER email12 | ∪ regex | 290 | 50 | 108 | 98.13% | 93.76% |
| XLM-R epoch 5 | naked | 298 | 42 | 67 | 98.78% | 89.36% |
| **NERGAL 1.1.2** | **∪ regex** | 326 | 23 | 80 | 98.65% | 96.49% |

Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex: 147/169 phone, 179/185 other PII. Exact-span precision 86.41%, recall 89.83%, F1 88.09%. The GLiNER rows use the original gold (10 characters of one span differ), and the two GLiNER ∪ regex rows use the 1.1.0 rules; the other rows use the amended gold.

Trained GLiNER, HerBERT-large, and XLM-R-large were also compared on this split (plot above). GLiNER’s 0.50 coverage lead is the precision collapse in that figure; no GLiNER or HerBERT operating point passed the content-preservation gate.

## Extra seeds

Historical seed-comparison results, before the 1.0.2 parser fix.

| Seed | Whole /354 | False chars | New false vs historical union |
|---|---:|---:|---:|
| 202609160 (selected weights) | 323 | 123 | 0 |
| 202609161 | 322 | 134 | 1 |
| 202609162 | 316 | 151 | 18 |

Seed 160 is the published weights. 161 and 162 were confirmation runs of the same recipe.

## Load

This repo is the PII island: `scrub_pii.py` plus `nergal.py`. `pipeline("token-classification")` will not match. Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[Telefon]` / `[PII]`. Text that already holds `[PII]` / `[Telefon]` is scrubbed as usual, but an identifier with a placeholder inside it or right before it can be missed.

```python
from pathlib import Path
from huggingface_hub import snapshot_download

root = Path(snapshot_download("SlayerLab/NERGAL"))
import sys
sys.path.insert(0, str(root))
from nergal import Nergal

nergal = Nergal.from_pretrained(root, local_files_only=True)
masked, counts = nergal.scrub(text)
```

For many texts, `nergal.scrub_many(texts)` (or `predict_many` for the raw model spans) batches windows across texts: about 2× the throughput of calling `scrub` in a loop on a CUDA GPU. `from_pretrained(..., dtype="float16")` casts the weights at load time on CUDA or MPS: another 1.9× on an RTX 4090, and 841-dev union numbers are unchanged, but scores are not bit-identical to float32 (0.95 spans can differ in rare cases). For a corpus, run 2–3 processes per GPU, because float16 inference is limited by CPU-side tokenization. Measurements are in `CHANGELOG.md`.

`hybrid.json` records version `1.1.2`, threshold 0.95, gap ids `250002` / `250003`, the 841-dev `eval` block (with its `gold_amendments`), and two weight hashes: `model_safetensors_sha256` for the published file and `source_checkpoint_sha256` for the training checkpoint it was packed from. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.

Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).