v2.1/model_card.md

---
language:
  - en
license: other
tags:
  - text-classification
  - phishing-detection
  - distilbert
base_model: distilbert-base-uncased
pipeline_tag: text-classification
---

# Model Card — PriorMail Phishing Detector v2.1

**Model ID:** `phishing/v2.1`
**Owner:** Faiz (PJK-GM095)
**Date:** 2026-06-19
**Git SHA:** `7cdeae7` (branch `feat/phishing-distilbert-v2`)
**Status:** In-distribution §8 gates **PASS**; real-world acceptance gate **FAILS on false-negative rate** (documented known-limitation below). Ship decision is deliberate, scoped, and pending sign-off from Insan + Syafiq — this is **not** an automatic promotion.

---

## What This Model Does

Binary email classifier: **legit (0)** vs **phishing (1)**.

Used by `prior-mail-backend` to flag potentially malicious emails. Intended as **one layer** of defense — see Known Limitations for why it should not be the sole line.

---

## Base Model

`distilbert-base-uncased` (~67M params).

Replaces v1.0's `bert-base-multilingual-cased`: the training data is English-only, and DistilBERT is ~3× smaller and faster on CPU (the backend runs CPU on Render), easing the 500 ms latency gate and the checkpoint-size constraint.

---

## Training Data

| Source | Role | Notes |
|---|---|---|
| `ealvaradob/phishing-dataset` (`texts`) | Phishing + benign legit | ~20,137 rows; ~62% benign |
| `insanar/prior-mail-priority` (`v4`) | Legit negatives | Product-distribution legit mail (transactional/banking) |
| Synthetic augmentation (`src/data/augment.py`) | Both classes | 600 rows/class, header-complete, seed 42 |

Enron (`emails.csv`) is **off** in v2 (it made v1 learn "non-Enron-style ⇒ phishing" → over-flagging). Legit pool downsampled to ~1.2× phishing; deduplicated on the first 200 chars of body before splitting to prevent leakage.

**After dedup:** ~16,669 rows (7,572 phishing / 9,097 legit)
**Split (stratified, seed 42):** train 11,667 / val 2,501 / test 2,501

The synthetic augmentation covers tactics the public corpora miss (BEC/CEO-fraud, fake invoice, brand- and IT-helpdesk credential harvest, delivery). It is **header-complete on both classes** so header *presence* cannot become a class proxy (the v1 leak).

---

## Input / Output Format

**Input** (`src/data/preprocess.build_phishing_input()`):
```
{sender display-name + domain} [SEP] {subject} [SEP] {body}
```
Text is cleaned with `keep_domains=True`: URL **host** and email **domain** are kept as signal (lookalike domains, odd TLDs) while the PII-bearing URL path/query and email local-part are masked. The `[SEP]` scaffolding is always present (header-less rows carry empty fields) — this is what makes the sender/subject reinstatement leak-safe.

**Output:**
- Softmax probability for class 1 (phishing), range [0, 1].
- Label `phishing` if probability ≥ threshold, else `legit`.
- **Threshold: 0.06**, stored in `threshold.json`. Selected on the validation set to maximize F-beta (β=2, recall-weighted — false negatives are the worst case) subject to precision ≥ 0.80; median of the optimal plateau.

---

## Training Config

See [`configs/phishing_v2.yaml`](../configs/phishing_v2.yaml). Key hyperparameters:

| Parameter | Value |
|---|---|
| Learning rate | 2e-5 (AdamW) |
| Batch size | 16 per device |
| Max sequence length | 512 |
| Epochs | 4 (early stopping, patience 2) |
| Early stopping metric | `f1_phishing` |
| Loss | Weighted cross-entropy, `phishing_class_multiplier=1.0` (threshold drives recall, not the loss) |
| Mixed precision | bf16 |
| Seed | 42 |

---

## Evaluation Results

### In-distribution test set (n=2,501) — §8 corpus gates

| Metric | Result | Gate | Status |
|---|---|---|---|
| Recall | **0.990** | ≥ 0.95 | PASS ✅ |
| Precision | **0.977** | ≥ 0.80 | PASS ✅ |
| Latency p95 (CPU) | **242 ms** | < 500 ms | PASS ✅ |
| F1 | 0.984 | — | — |

Confusion matrix (test):

|  | Predicted Legit | Predicted Phishing |
|---|---|---|
| **Actual Legit** | 1,339 (TN) | 26 (FP) |
| **Actual Phishing** | 11 (FN) | 1,125 (TP) |

> The test split is drawn from the **same corpora** as training, so it measures in-distribution fit, not real-world behaviour (v1.0 passed these yet over-flagged real mail). The acceptance set below is the real-world judge.

### Real-world acceptance set (36 hand-curated `.eml`, 18 legit / 18 phishing)

| Metric | Result | Gate | Status |
|---|---|---|---|
| False-positive rate | **0.00** (0/18) | ≤ 0.20 | PASS ✅ |
| False-negative rate | **0.33** (6/18) | ≤ 0.05 | **FAIL ❌** |
| Accuracy | 0.833 | — | — |

- **Zero false positives** — including deliberately "scary but legit" mail (a real new-sign-in security alert, bank statement, password-reset notice, CI/Slack/GitHub notifications). The v1/v2 over-flagging problem is resolved.
- **Caught (12/18):** credential harvest (PayPal, IT-helpdesk), BEC/CEO-fraud, fake invoice, bank/card suspension, lottery, package delivery, tax refund, crypto wallet, job-offer scam, voicemail lure.
- **Missed (6/18), all scored ≈0.00:** phishing that **mimics a routine SaaS/workflow notification** — DocuSign signature request, MFA-reset notice, payroll direct-deposit redirect, shared-document lure, subscription-renewal scare, and an Office 365 lookalike-domain login.

---

## Known Limitations

- **Workflow-mimicking phishing (primary gap).** The model learned "phishing = urgent threat + suspicious URL." Phishing that impersonates a *calm, routine* notification (DocuSign/MFA/payroll/shared-document/subscription) whose only tell is a lookalike sender domain is scored confidently legit. This is the **ceiling of synthetic-template augmentation** — the model reliably catches tactics it was trained on and misses novel ones. Closing it requires **diverse real phishing data** (e.g. Nazario / PhishTank email corpora), not more synthetic templates, and/or a **header/auth-signal layer (SPF/DKIM/DMARC, display-name↔domain mismatch) at the backend** — which is the right home for sender-spoofing detection.
- **Acceptance FN gate fails (0.33 > 0.05).** Per CLAUDE.md §8/§12 this is *not* a clean promotion. It is shipped as a scoped candidate with this limitation documented; do **not** treat the model as a sole line of defense.
- **English only.** Indonesian-language phishing is not represented and not validated.
- **Adversarial robustness** (obfuscated URLs, Unicode/homoglyph tricks beyond simple lookalikes) is not systematically evaluated.

---

## Artifacts

| File | Description |
|---|---|
| `model.safetensors` | Fine-tuned weights (~256 MB) |
| `config.json` | HuggingFace model config (`id2label`: 0=legit, 1=phishing) |
| `tokenizer.json`, `tokenizer_config.json`, `vocab.txt` | Tokenizer (distilbert-base-uncased) |
| `threshold.json` | `{"threshold": 0.06}` |
| `training_config.yaml` | Full training config + git SHA + timestamp |
| `eval_report.json` | In-distribution test metrics + confusion matrix + latency |
| `acceptance_report.json` | 36-file real-world acceptance results (per-file table) |
| `val_metrics.json` | Validation metrics at best epoch |

---

## Changelog

| Version | Date | Notes |
|---|---|---|
| v1.0 | 2026-06-13 | Initial release. mBERT. Passed corpus gates; over-flagged real mail at inference. |
| v2.1 | 2026-06-19 | DistilBERT rework. Domain-aware cleaning (A1), header-complete synthetic augmentation + sender/subject input (B1/B2), FN-averse threshold selection, real-world `.eml` acceptance gate. Corpus gates pass, FP eliminated (0%); documented FN gap on workflow-mimicking phishing. |

> Reproduce: `make data-phishing && make train-phishing config=configs/phishing_v2.yaml && make eval-phishing config=configs/phishing_v2.yaml` then `python -m src.eval.acceptance_phishing --config configs/phishing_v2.yaml` (git `7cdeae7`, seed 42).