Improve model card, metadata, metrics, and usage example

README.md

@@ -1,120 +1,264 @@
 ---
 language: en
 license: apache-2.0
+library_name: transformers
+pipeline_tag: token-classification
+base_model: microsoft/deberta-v3-base
 tags:
   - token-classification
   - ner
+  - phi
+  - pii
   - privacy
   - healthcare
   - deidentification
   - security
   - compliance
-pipeline_tag: token-classification
-library_name: transformers
+  - synthetic-data
+  - deberta-v3
+widget:
+  - text: "Patient John Smith (MRN: 001-23-4567) visited Boston Medical Center on 12/19/2025."
+  - text: "Discharge summary reviewed by Dr. Emily Chen at Riverfront Clinic on 03/07/2025."
+model-index:
+  - name: phi-span-detector-deberta-v3
+    results:
+      - task:
+          type: token-classification
+          name: Token Classification
+        dataset:
+          name: Synthetic PHI span test set
+          type: synthetic
+        metrics:
+          - name: Micro F1
+            type: f1
+            value: 0.6523
+          - name: Micro Precision
+            type: precision
+            value: 0.6657
+          - name: Micro Recall
+            type: recall
+            value: 0.6394
+          - name: Macro F1
+            type: f1
+            value: 0.6362
 ---
 
-# PHI Span Detector (BIO NER) — Synthetic
+# PHI Span Detector (BIO NER) - Synthetic
 
-This model detects **Protected Health Information (PHI)** spans in clinical-note-like text and log-like text using **BIO tagging** (token classification). It is intended to power **deterministic redaction** and **zero-trust logging guardrails**.
+`phi-span-detector-deberta-v3` is a DeBERTa v3 token-classification model for detecting Protected Health Information (PHI) spans in clinical-note-like text and log-like text using BIO tagging.
 
-## PHI Types
+It is designed for privacy tooling workflows such as:
 
-The model predicts spans for the following categories:
+- deterministic redaction pipelines
+- pre-log and post-log PHI guardrails
+- research prototypes for de-identification
 
-- **NAME**
-- **DATE**
-- **AGE**
-- **PHONE**
-- **EMAIL**
-- **ADDRESS**
-- **ID** (e.g., MRN/account/record IDs)
-- **PROVIDER**
-- **FACILITY**
-- **LOCATION**
+Recommended pipeline:
 
-Output is BIO-formatted per token (e.g., `B-NAME`, `I-NAME`, …).
+1. detect PHI spans
+2. apply deterministic redaction
+3. run a secondary leak-check gate before downstream use
 
----
+Companion model:
+[`bharathjanumpally/phi-leak-checker-deberta-v3`](https://huggingface.co/bharathjanumpally/phi-leak-checker-deberta-v3)
 
-## How it works
+## Model at a glance
 
-This is a **token-classification** model trained on **synthetic** examples to keep the project openly shareable:
+- Task: token classification
+- Architecture: `DebertaV2ForTokenClassification`
+- Base model: `microsoft/deberta-v3-base`
+- Max sequence length: 512
+- Labeling scheme: BIO
+- Training data: synthetic text only
 
-1. Synthetic clinical notes and log lines are generated using templates.
-2. PHI-like fields are inserted (names, IDs, phone numbers, dates, addresses, etc.).
-3. Gold labels are produced automatically as character spans and converted to BIO token labels.
+## PHI label set
 
-This produces clean supervision without using real patient data.
+The model predicts the following entity families:
 
----
+| Label | Meaning |
+| --- | --- |
+| `NAME` | patient or person names |
+| `DATE` | visit dates, birth dates, service dates |
+| `AGE` | age mentions that may be identifying in context |
+| `PHONE` | phone and callback numbers |
+| `EMAIL` | email addresses |
+| `ADDRESS` | street or mailing addresses |
+| `ID` | MRN, account, encounter, record, or similar identifiers |
+| `PROVIDER` | clinician or provider names |
+| `FACILITY` | hospitals, clinics, centers, departments |
+| `LOCATION` | city, state, and other place references |
 
-## Intended Use
+Token-level outputs use BIO labels from the model config:
 
-✅ **Appropriate uses**
-- PHI span detection for research prototypes
-- Pre-log / post-log redaction guardrails
-- De-identification pipelines when paired with deterministic redaction
+`O`, `B-*`, and `I-*` across the ten PHI families above.
 
-❌ **Not intended for**
-- Medical diagnosis or treatment advice
-- Sole control for compliance (HIPAA/GDPR) decisions
-- High-stakes production usage without additional safeguards and evaluation
+## How the training data was built
 
-**Recommended pipeline:** Detect spans → deterministic redaction → secondary leak-check gate.
+This model was trained on synthetic examples to keep the project openly shareable.
 
----
+High-level training recipe:
 
-## Limitations
+1. Generate synthetic clinical notes and log-like text with templates.
+2. Insert PHI-like fields such as names, dates, IDs, facilities, phone numbers, and addresses.
+3. Convert gold character spans into BIO token labels for token classification.
 
-- Trained on **synthetic** text: real-world clinical documentation can include unseen formats and edge cases.
-- May over-redact (false positives) on numeric identifiers or location-like strings.
-- May miss rare PHI patterns not represented in synthetic templates.
+This provides clean supervision without exposing real patient data, but it also means real-world formatting and writing styles may differ from training-time distributions.
 
-If using in a real system, evaluate on your organization’s internal test set and consider adding:
-- regex backstops (email/phone/date patterns)
-- human-in-the-loop review for flagged cases
-- a secondary “PHI leak checker” model
+## Evaluation
 
----
+The repository includes a full [`seqeval_report.txt`](./seqeval_report.txt). Key held-out results from that report are summarized below.
+
+### Overall metrics
+
+| Metric | Value |
+| --- | ---: |
+| Micro precision | 0.6657 |
+| Micro recall | 0.6394 |
+| Micro F1 | 0.6523 |
+| Macro precision | 0.6583 |
+| Macro recall | 0.6224 |
+| Macro F1 | 0.6362 |
+| Weighted F1 | 0.6495 |
+
+### Per-label metrics
+
+| Label | Precision | Recall | F1 | Support |
+| --- | ---: | ---: | ---: | ---: |
+| ADDRESS | 0.6652 | 0.6481 | 0.6565 | 233 |
+| AGE | 0.6758 | 0.3834 | 0.4893 | 386 |
+| DATE | 0.6553 | 0.6492 | 0.6522 | 1297 |
+| EMAIL | 0.6474 | 0.6455 | 0.6465 | 347 |
+| FACILITY | 0.6320 | 0.6494 | 0.6406 | 656 |
+| ID | 0.6652 | 0.6519 | 0.6585 | 451 |
+| LOCATION | 0.6600 | 0.6600 | 0.6600 | 350 |
+| NAME | 0.7810 | 0.7802 | 0.7806 | 1001 |
+| PHONE | 0.5358 | 0.5025 | 0.5186 | 595 |
+| PROVIDER | 0.6652 | 0.6537 | 0.6594 | 231 |
+
+### Interpretation
+
+- Strongest label in the current report: `NAME`
+- Weakest labels in the current report: `PHONE` and `AGE`
+- The model is usable as a PHI span detector for research and tooling, but it should be paired with deterministic rules and internal evaluation before higher-stakes deployment
+
+## Intended use
+
+Appropriate uses:
+
+- PHI span detection in research prototypes
+- de-identification pipelines when paired with deterministic redaction
+- zero-trust logging guardrails
+- preprocessing before a secondary PHI leak checker
+
+Not intended for:
+
+- medical diagnosis or treatment advice
+- sole control for HIPAA, GDPR, or other compliance decisions
+- unsupervised high-stakes production usage without internal validation
+
+## Limitations and failure modes
+
+- The model was trained on synthetic text, so real clinical documentation may include unseen abbreviations, formatting quirks, shorthand, OCR noise, and edge cases.
+- Numeric strings may be over-flagged when they resemble IDs, dates, or phone numbers.
+- Some rare PHI patterns may be missed if they were not well represented in the synthetic templates.
+- Partial tokens and tokenizer boundary effects can require careful post-processing in downstream systems.
+- Label performance is uneven; current metrics suggest extra caution around `PHONE` and `AGE`.
+
+Recommended mitigations:
+
+- add regex backstops for structured entities like email, phone, and date
+- apply deterministic placeholder redaction after detection
+- run a second PHI leak-check model before downstream release
+- evaluate on an internal, policy-approved test set that matches your real document style
+- keep a human-review path for ambiguous or high-risk content
 
 ## Usage
 
-### 1) Transformers token-classification pipeline
+### Transformers pipeline
+
 ```python
 from transformers import pipeline
 
 ner = pipeline(
     "token-classification",
-    model="bharathja/phi-span-detector-deberta-v3",
-    aggregation_strategy="simple"
+    model="bharathjanumpally/phi-span-detector-deberta-v3",
+    aggregation_strategy="simple",
+)
+
+text = (
+    "Patient John Smith (MRN: 001-23-4567) visited "
+    "Boston Medical Center on 12/19/2025."
 )
 
-text = "Patient John Smith (MRN: 001-23-4567) visited Boston Medical Center on 12/19/2025."
 print(ner(text))
 ```
-### 2) Deterministic redaction (recommended)
 
-Use detected spans to redact with placeholders such as [NAME], [ID], [DATE], etc.
-(See companion project: PHI Guardrails.)
+### AutoModel and AutoTokenizer
+
+```python
+from transformers import AutoModelForTokenClassification, AutoTokenizer, pipeline
+
+model_id = "bharathjanumpally/phi-span-detector-deberta-v3"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForTokenClassification.from_pretrained(model_id)
+
+ner = pipeline(
+    "token-classification",
+    model=model,
+    tokenizer=tokenizer,
+    aggregation_strategy="simple",
+)
+
+print(ner("Call Jane Doe at 617-555-0182 before 04/14/2025."))
+```
+
+### Deterministic redaction example
 
-Output Schema (recommended)
 ```python
-A practical production-friendly span format:
+from transformers import pipeline
+
+ner = pipeline(
+    "token-classification",
+    model="bharathjanumpally/phi-span-detector-deberta-v3",
+    aggregation_strategy="simple",
+)
 
+text = (
+    "Patient John Smith (MRN: 001-23-4567) visited "
+    "Boston Medical Center on 12/19/2025."
+)
+
+spans = ner(text)
+
+redacted = text
+for item in sorted(spans, key=lambda x: x["start"], reverse=True):
+    label = item["entity_group"]
+    redacted = redacted[: item["start"]] + f"[{label}]" + redacted[item["end"] :]
+
+print(spans)
+print(redacted)
+```
+
+### Example output schema
+
+For downstream systems, a practical span schema is:
+
+```json
 [
   {"start": 8, "end": 18, "label": "NAME", "score": 0.97},
   {"start": 25, "end": 36, "label": "ID", "score": 0.94},
-  {"start": 68, "end": 78, "label": "FACILITY", "score": 0.91},
-  {"start": 82, "end": 92, "label": "DATE", "score": 0.89}
+  {"start": 45, "end": 66, "label": "FACILITY", "score": 0.91},
+  {"start": 70, "end": 80, "label": "DATE", "score": 0.89}
 ]
 ```
 
-### Safety & Privacy
+## Safety and privacy
 
-This model is trained on synthetic data and is published for research and tooling purposes.
-Do not upload real PHI to public endpoints or demos. Use private infrastructure for real deployments.
-```python
-Citation
+This model was trained on synthetic data and is published for research and tooling purposes. Do not send real PHI to public demos or public inference endpoints. Use private infrastructure, access controls, and organization-approved evaluation workflows for real deployments.
+
+## Citation
+
+```bibtex
 @misc{janumpally_phi_span_detector_2025,
   title        = {PHI Span Detector (Synthetic)},
   author       = {Bharath Kumar Reddy Janumpally},
@@ -122,5 +266,8 @@ Citation
   publisher    = {Hugging Face},
   howpublished = {Model on Hugging Face}
 }
+```
+
+## Contact
 
-````
+If you use this model in a serious workflow, validate it against your own internal test cases and document the operating policy around false positives, false negatives, and escalation paths.

example_redaction.py

@@ -0,0 +1,34 @@
+from transformers import pipeline
+
+
+MODEL_ID = "bharathjanumpally/phi-span-detector-deberta-v3"
+
+
+def redact_text(text: str) -> tuple[list[dict], str]:
+    ner = pipeline(
+        "token-classification",
+        model=MODEL_ID,
+        aggregation_strategy="simple",
+    )
+    spans = ner(text)
+
+    redacted = text
+    for item in sorted(spans, key=lambda x: x["start"], reverse=True):
+        label = item["entity_group"]
+        redacted = redacted[: item["start"]] + f"[{label}]" + redacted[item["end"] :]
+
+    return spans, redacted
+
+
+if __name__ == "__main__":
+    sample = (
+        "Patient John Smith (MRN: 001-23-4567) visited "
+        "Boston Medical Center on 12/19/2025."
+    )
+    spans, redacted = redact_text(sample)
+    print("Spans:")
+    for span in spans:
+        print(span)
+    print()
+    print("Redacted:")
+    print(redacted)