Publish phishing v2.1

v2.1/config.json

@@ -0,0 +1,35 @@
+{
+  "activation": "gelu",
+  "architectures": [
+    "DistilBertForSequenceClassification"
+  ],
+  "attention_dropout": 0.1,
+  "bos_token_id": null,
+  "dim": 768,
+  "dropout": 0.1,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_dim": 3072,
+  "id2label": {
+    "0": "legit",
+    "1": "phishing"
+  },
+  "initializer_range": 0.02,
+  "label2id": {
+    "legit": 0,
+    "phishing": 1
+  },
+  "max_position_embeddings": 512,
+  "model_type": "distilbert",
+  "n_heads": 12,
+  "n_layers": 6,
+  "pad_token_id": 0,
+  "qa_dropout": 0.1,
+  "seq_classif_dropout": 0.2,
+  "sinusoidal_pos_embds": false,
+  "tie_weights_": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "vocab_size": 30522
+}

v2.1/eval_report.json

@@ -0,0 +1,36 @@
+{
+  "checkpoint": "checkpoints/phishing_v2",
+  "test_split": "data/processed/phishing/test",
+  "metrics": {
+    "recall_phishing": 0.9903169014084507,
+    "precision_phishing": 0.9774109470026064,
+    "f1_phishing": 0.9838216003498033,
+    "confusion_matrix": [
+      [
+        1339,
+        26
+      ],
+      [
+        11,
+        1125
+      ]
+    ],
+    "threshold": 0.06,
+    "n_test": 2501,
+    "n_phishing_test": 1136,
+    "n_legit_test": 1365,
+    "p95_ms": 242.0971423996889,
+    "latency": {
+      "mean_ms": 100.66626771500523,
+      "p50_ms": 59.67643049984872,
+      "p95_ms": 242.0971423996889,
+      "n": 200.0
+    }
+  },
+  "gates": {
+    "recall": true,
+    "precision": true,
+    "latency_p95": true
+  },
+  "all_passed": true
+}

v2.1/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7692afc2685e34b5736a8af98aa889929c6c4a7ce2850adfce009d32e82ecfc5
+size 267832560

v2.1/model_card.md

@@ -0,0 +1,153 @@
+---
+language:
+  - en
+license: other
+tags:
+  - text-classification
+  - phishing-detection
+  - distilbert
+base_model: distilbert-base-uncased
+pipeline_tag: text-classification
+---
+
+# Model Card — PriorMail Phishing Detector v2.1
+
+**Model ID:** `phishing/v2.1`
+**Owner:** Faiz (PJK-GM095)
+**Date:** 2026-06-19
+**Git SHA:** `7cdeae7` (branch `feat/phishing-distilbert-v2`)
+**Status:** In-distribution §8 gates **PASS**; real-world acceptance gate **FAILS on false-negative rate** (documented known-limitation below). Ship decision is deliberate, scoped, and pending sign-off from Insan + Syafiq — this is **not** an automatic promotion.
+
+---
+
+## What This Model Does
+
+Binary email classifier: **legit (0)** vs **phishing (1)**.
+
+Used by `prior-mail-backend` to flag potentially malicious emails. Intended as **one layer** of defense — see Known Limitations for why it should not be the sole line.
+
+---
+
+## Base Model
+
+`distilbert-base-uncased` (~67M params).
+
+Replaces v1.0's `bert-base-multilingual-cased`: the training data is English-only, and DistilBERT is ~3× smaller and faster on CPU (the backend runs CPU on Render), easing the 500 ms latency gate and the checkpoint-size constraint.
+
+---
+
+## Training Data
+
+| Source | Role | Notes |
+|---|---|---|
+| `ealvaradob/phishing-dataset` (`texts`) | Phishing + benign legit | ~20,137 rows; ~62% benign |
+| `insanar/prior-mail-priority` (`v4`) | Legit negatives | Product-distribution legit mail (transactional/banking) |
+| Synthetic augmentation (`src/data/augment.py`) | Both classes | 600 rows/class, header-complete, seed 42 |
+
+Enron (`emails.csv`) is **off** in v2 (it made v1 learn "non-Enron-style ⇒ phishing" → over-flagging). Legit pool downsampled to ~1.2× phishing; deduplicated on the first 200 chars of body before splitting to prevent leakage.
+
+**After dedup:** ~16,669 rows (7,572 phishing / 9,097 legit)
+**Split (stratified, seed 42):** train 11,667 / val 2,501 / test 2,501
+
+The synthetic augmentation covers tactics the public corpora miss (BEC/CEO-fraud, fake invoice, brand- and IT-helpdesk credential harvest, delivery). It is **header-complete on both classes** so header *presence* cannot become a class proxy (the v1 leak).
+
+---
+
+## Input / Output Format
+
+**Input** (`src/data/preprocess.build_phishing_input()`):
+```
+{sender display-name + domain} [SEP] {subject} [SEP] {body}
+```
+Text is cleaned with `keep_domains=True`: URL **host** and email **domain** are kept as signal (lookalike domains, odd TLDs) while the PII-bearing URL path/query and email local-part are masked. The `[SEP]` scaffolding is always present (header-less rows carry empty fields) — this is what makes the sender/subject reinstatement leak-safe.
+
+**Output:**
+- Softmax probability for class 1 (phishing), range [0, 1].
+- Label `phishing` if probability ≥ threshold, else `legit`.
+- **Threshold: 0.06**, stored in `threshold.json`. Selected on the validation set to maximize F-beta (β=2, recall-weighted — false negatives are the worst case) subject to precision ≥ 0.80; median of the optimal plateau.
+
+---
+
+## Training Config
+
+See [`configs/phishing_v2.yaml`](../configs/phishing_v2.yaml). Key hyperparameters:
+
+| Parameter | Value |
+|---|---|
+| Learning rate | 2e-5 (AdamW) |
+| Batch size | 16 per device |
+| Max sequence length | 512 |
+| Epochs | 4 (early stopping, patience 2) |
+| Early stopping metric | `f1_phishing` |
+| Loss | Weighted cross-entropy, `phishing_class_multiplier=1.0` (threshold drives recall, not the loss) |
+| Mixed precision | bf16 |
+| Seed | 42 |
+
+---
+
+## Evaluation Results
+
+### In-distribution test set (n=2,501) — §8 corpus gates
+
+| Metric | Result | Gate | Status |
+|---|---|---|---|
+| Recall | **0.990** | ≥ 0.95 | PASS ✅ |
+| Precision | **0.977** | ≥ 0.80 | PASS ✅ |
+| Latency p95 (CPU) | **242 ms** | < 500 ms | PASS ✅ |
+| F1 | 0.984 | — | — |
+
+Confusion matrix (test):
+
+|  | Predicted Legit | Predicted Phishing |
+|---|---|---|
+| **Actual Legit** | 1,339 (TN) | 26 (FP) |
+| **Actual Phishing** | 11 (FN) | 1,125 (TP) |
+
+> The test split is drawn from the **same corpora** as training, so it measures in-distribution fit, not real-world behaviour (v1.0 passed these yet over-flagged real mail). The acceptance set below is the real-world judge.
+
+### Real-world acceptance set (36 hand-curated `.eml`, 18 legit / 18 phishing)
+
+| Metric | Result | Gate | Status |
+|---|---|---|---|
+| False-positive rate | **0.00** (0/18) | ≤ 0.20 | PASS ✅ |
+| False-negative rate | **0.33** (6/18) | ≤ 0.05 | **FAIL ❌** |
+| Accuracy | 0.833 | — | — |
+
+- **Zero false positives** — including deliberately "scary but legit" mail (a real new-sign-in security alert, bank statement, password-reset notice, CI/Slack/GitHub notifications). The v1/v2 over-flagging problem is resolved.
+- **Caught (12/18):** credential harvest (PayPal, IT-helpdesk), BEC/CEO-fraud, fake invoice, bank/card suspension, lottery, package delivery, tax refund, crypto wallet, job-offer scam, voicemail lure.
+- **Missed (6/18), all scored ≈0.00:** phishing that **mimics a routine SaaS/workflow notification** — DocuSign signature request, MFA-reset notice, payroll direct-deposit redirect, shared-document lure, subscription-renewal scare, and an Office 365 lookalike-domain login.
+
+---
+
+## Known Limitations
+
+- **Workflow-mimicking phishing (primary gap).** The model learned "phishing = urgent threat + suspicious URL." Phishing that impersonates a *calm, routine* notification (DocuSign/MFA/payroll/shared-document/subscription) whose only tell is a lookalike sender domain is scored confidently legit. This is the **ceiling of synthetic-template augmentation** — the model reliably catches tactics it was trained on and misses novel ones. Closing it requires **diverse real phishing data** (e.g. Nazario / PhishTank email corpora), not more synthetic templates, and/or a **header/auth-signal layer (SPF/DKIM/DMARC, display-name↔domain mismatch) at the backend** — which is the right home for sender-spoofing detection.
+- **Acceptance FN gate fails (0.33 > 0.05).** Per CLAUDE.md §8/§12 this is *not* a clean promotion. It is shipped as a scoped candidate with this limitation documented; do **not** treat the model as a sole line of defense.
+- **English only.** Indonesian-language phishing is not represented and not validated.
+- **Adversarial robustness** (obfuscated URLs, Unicode/homoglyph tricks beyond simple lookalikes) is not systematically evaluated.
+
+---
+
+## Artifacts
+
+| File | Description |
+|---|---|
+| `model.safetensors` | Fine-tuned weights (~256 MB) |
+| `config.json` | HuggingFace model config (`id2label`: 0=legit, 1=phishing) |
+| `tokenizer.json`, `tokenizer_config.json`, `vocab.txt` | Tokenizer (distilbert-base-uncased) |
+| `threshold.json` | `{"threshold": 0.06}` |
+| `training_config.yaml` | Full training config + git SHA + timestamp |
+| `eval_report.json` | In-distribution test metrics + confusion matrix + latency |
+| `acceptance_report.json` | 36-file real-world acceptance results (per-file table) |
+| `val_metrics.json` | Validation metrics at best epoch |
+
+---
+
+## Changelog
+
+| Version | Date | Notes |
+|---|---|---|
+| v1.0 | 2026-06-13 | Initial release. mBERT. Passed corpus gates; over-flagged real mail at inference. |
+| v2.1 | 2026-06-19 | DistilBERT rework. Domain-aware cleaning (A1), header-complete synthetic augmentation + sender/subject input (B1/B2), FN-averse threshold selection, real-world `.eml` acceptance gate. Corpus gates pass, FP eliminated (0%); documented FN gap on workflow-mimicking phishing. |
+
+> Reproduce: `make data-phishing && make train-phishing config=configs/phishing_v2.yaml && make eval-phishing config=configs/phishing_v2.yaml` then `python -m src.eval.acceptance_phishing --config configs/phishing_v2.yaml` (git `7cdeae7`, seed 42).

v2.1/threshold.json

@@ -0,0 +1,3 @@
+{
+  "threshold": 0.06
+}

v2.1/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

v2.1/training_config.yaml

@@ -0,0 +1,76 @@
+# Phishing detector — v2 config (DistilBERT rework).
+#
+# v2.1 update (2026-06-19): the v2 run passed the §8 in-distribution gates but
+# missed half the real .eml acceptance phishing (FN rate 0.50). Fixes landed in
+# the data/preprocess layer (no hyperparameter change needed):
+#   A1 — clean_text(keep_domains=True): URL host + email domain kept as signal
+#        (lookalike domains / odd TLDs) instead of an opaque [URL].
+#   B1 — synthetic augmentation: prepare adds 600 rows/class (header-complete,
+#        both classes) covering BEC / fake-invoice / brand- AND IT-helpdesk
+#        credential tactics the corpora miss. Tune via --augmentation-size.
+#   B2 — input is now {sender_domain} [SEP] {subject} [SEP] {body} (was body-only);
+#        leak-safe because the [SEP] scaffolding is always present and the
+#        augmentation gives BOTH classes realistic headers.
+#   Real-world promotion is now also gated on the .eml acceptance set
+#   (src/eval/acceptance_phishing.py), not just the in-distribution §8 metrics.
+#
+# Why v2 (see feat/phishing-distilbert-v2 analysis, 2026-06-18):
+#   v1.0 passed test gates (recall 0.970 / precision 0.928) but over-flagged on
+#   real .eml at inference. Root causes addressed here:
+#     #1 Base model  : distilbert-base-uncased — on-spec (ML_PIPELINE.md §3, mBERT
+#                      dropped: data is English-only; DistilBERT is faster on CPU
+#                      and ~3× smaller, easing the 500 ms gate + checkpoint size).
+#     #4 Class weight : multiplier dropped 3.0 -> 1.0. v1 stacked balanced (inverse-
+#                      frequency) AND a 3× multiplier => ~6× gradient pressure on
+#                      phishing, saturating its probability for almost any input.
+#                      Pure inverse-frequency now; the THRESHOLD does the recall
+#                      work, not the loss.
+#     #5 Early stop   : monitor f1_phishing, not recall_phishing. Monitoring raw
+#                      recall picked the most trigger-happy checkpoint; F1 balances
+#                      recall against precision. Recall is still guaranteed by the
+#                      eval-time threshold + gate.
+#   (#2 legit diversity + #3 body-only input are in loaders.py / preprocess.py.)
+#
+# Run via:
+#   make data-phishing
+#   make train-phishing config=configs/phishing_v2.yaml
+
+model_type: phishing
+model_name: distilbert-base-uncased   # §3 decision: English-only data, on-spec
+
+dataset: ealvaradob/phishing-dataset       # primary phishing training data (HF)
+legit_source: emails.csv                   # Enron corpus — negative class source
+
+num_labels: 2   # legit (0) | phishing (1)  — matches PHISHING_LABELS in constants.py
+
+hyperparameters:
+  learning_rate: 2.0e-5         # AdamW (mirrors priority protocol, ML_PIPELINE.md §3)
+  weight_decay: 0.01
+  batch_size: 16
+  gradient_accumulation_steps: 1
+  num_epochs: 4                 # 3–5 with early stopping
+  warmup_ratio: 0.10
+  lr_scheduler_type: linear
+  max_seq_length: 512
+  # Early stopping on F1 — balances recall vs precision (fix #5). Recall is still
+  # enforced by the post-training threshold + the >=0.95 eval gate.
+  early_stopping_metric: f1_phishing
+  early_stopping_patience: 2
+  class_weights: balanced                # inverse-frequency base weights from train dist
+  phishing_class_multiplier: 1.0         # fix #4: no extra stacking — let threshold drive recall
+  mixed_precision: bf16                  # falls back to fp16 / none as needed
+
+seed: 42
+
+output_dir: checkpoints/phishing_v2
+
+wandb:
+  project: priormail
+  tags:
+    - model:phishing
+    - stage:exp
+
+# --- run provenance (auto-recorded) ---
+_git_sha: 7cdeae7
+_git_dirty: False
+_trained_at: 2026-06-19T03:00:44