Publish phishing v1.0

v1.0/config.json

@@ -0,0 +1,42 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "legit",
+    "1": "phishing"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "label2id": {
+    "legit": 0,
+    "phishing": 1
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "type_vocab_size": 2,
+  "use_cache": false,
+  "vocab_size": 119547
+}

v1.0/eval_report.json

@@ -0,0 +1,36 @@
+{
+  "checkpoint": "checkpoints\\phishing_v1",
+  "test_split": "data\\processed\\phishing\\test",
+  "metrics": {
+    "recall_phishing": 0.9695238095238096,
+    "precision_phishing": 0.9279854147675478,
+    "f1_phishing": 0.9482999534233815,
+    "confusion_matrix": [
+      [
+        4593,
+        79
+      ],
+      [
+        32,
+        1018
+      ]
+    ],
+    "threshold": 0.95,
+    "n_test": 5722,
+    "n_phishing_test": 1050,
+    "n_legit_test": 4672,
+    "p95_ms": 434.70457002258627,
+    "latency": {
+      "mean_ms": 239.72067400012747,
+      "p50_ms": 220.47114999440964,
+      "p95_ms": 434.70457002258627,
+      "n": 200.0
+    }
+  },
+  "gates": {
+    "recall": true,
+    "precision": true,
+    "latency_p95": true
+  },
+  "all_passed": true
+}

v1.0/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3833500e476c2272e55bf008849e8afad126520073c308297c4af742a656fde5
+size 711443432

v1.0/model_card.md

@@ -0,0 +1,143 @@
+# Model Card — PriorMail Phishing Detector v1.0
+
+**Model ID:** `phishing/v1.0`
+**Owner:** Faiz (PJK-GM095)
+**Date:** 2026-06-13
+**Status:** Eval gates passed — pending export approval from Insan + Syafiq
+
+---
+
+## What This Model Does
+
+Binary email classifier: **legit (0)** vs **phishing (1)**.
+
+Used by `prior-mail-backend` to flag potentially malicious emails before they reach the user's inbox. Runs at inference time on every incoming email.
+
+---
+
+## Base Model
+
+`bert-base-multilingual-cased` (Google, 178M params)
+
+Chosen over IndoBERT because the primary training data is English-dominant (~98% EN). mBERT handles English well and retains multilingual capability for future Indonesian phishing data.
+
+---
+
+## Training Data
+
+| Source | Role | Rows (after dedup) |
+|---|---|---|
+| `ealvaradob/phishing-dataset` subset `texts` | Phishing + some legit | ~20,137 |
+| Enron corpus (`emails.csv`) | Legit negative class | ~20,000 sampled |
+
+**Total after deduplication:** ~38,000 rows  
+**Train / Val / Test split:** 75 / 12.5 / 12.5 (stratified by label)
+
+Deduplication applied on the first 200 chars of body before splitting to prevent leakage.
+
+**Note:** Dataset is almost entirely English. Indonesian phishing emails are not well-represented. Performance on Indonesian-language phishing may be lower — to be addressed in v1.1 with internal labeled data.
+
+---
+
+## Training Config
+
+See [`configs/phishing_v1.yaml`](../configs/phishing_v1.yaml) for the full reproducible config.
+
+Key hyperparameters:
+
+| Parameter | Value |
+|---|---|
+| Learning rate | 2e-5 |
+| Batch size | 16 per device |
+| Max sequence length | 512 |
+| Epochs | 4 (early stopped at 3) |
+| Warmup ratio | 0.10 |
+| Loss | Weighted cross-entropy (phishing_class_multiplier=3.0) |
+| Early stopping metric | `recall_phishing` (patience=2) |
+| Mixed precision | bf16 |
+| Seed | 42 |
+
+---
+
+## Evaluation Results
+
+### Threshold Selection (Validation Set)
+
+Threshold selected on val set to maximize precision while keeping recall ≥ 0.95.
+
+**Threshold: 0.95**
+
+| Metric | Val Set |
+|---|---|
+| Recall | 0.981 |
+| Precision | 0.933 |
+
+### Final Results (Test Set, n=5,722)
+
+| Metric | Result | Gate | Status |
+|---|---|---|---|
+| Recall | **0.970** | ≥ 0.95 | PASS ✅ |
+| Precision | **0.928** | ≥ 0.80 | PASS ✅ |
+| Latency p95 (CPU) | **435 ms** | < 500 ms | PASS ✅ |
+| F1 | 0.948 | — | — |
+
+### Confusion Matrix (Test Set)
+
+|  | Predicted Legit | Predicted Phishing |
+|---|---|---|
+| **Actual Legit** | 4,593 (TN) | 79 (FP) |
+| **Actual Phishing** | 32 (FN) | 1,018 (TP) |
+
+False negatives (missed phishing): **32 out of 1,050** — 3.0% miss rate.  
+False positives (false alarms): **79 out of 4,672** — 1.7% false alarm rate.
+
+---
+
+## Input / Output Format
+
+**Input** (assembled by `src/data/preprocess.build_phishing_input()`):
+```
+FROM: {sender_email} [SEP] SUBJECT: {subject} [SEP] BODY: {body}
+```
+
+If sender or subject are unavailable (e.g. plain-body-only sources), those fields are empty strings. The model still performs well on body-only input — the HF training data was ~98% body-only.
+
+**Output:**
+- Raw: softmax probability for class 1 (phishing), range [0, 1]
+- Final label: `phishing` if probability ≥ 0.95, else `legit`
+- Threshold stored in `threshold.json` alongside the checkpoint
+
+---
+
+## Known Limitations
+
+- **Language:** Trained primarily on English data. Lower recall expected on Indonesian-language phishing. Mitigated by mBERT's multilingual pretraining, but not validated.
+- **Sender signal weak:** ~98% of HF phishing training samples had no RFC 2822 headers — sender email is not a reliable feature for this version. Enron legit samples do have sender headers, which may introduce a small asymmetry.
+- **Adversarial robustness:** Not evaluated against adversarial phishing (obfuscated URLs, lookalike domains, Unicode tricks). Planned for v1.1.
+- **Domain shift:** Trained on public phishing corpora. Real-world phishing targeting Indonesian corporate users may have different patterns.
+
+---
+
+## Artifacts
+
+| File | Description |
+|---|---|
+| `model.safetensors` | Fine-tuned weights (best checkpoint, step 835 / epoch 1) |
+| `config.json` | HuggingFace model config |
+| `tokenizer.json`, `tokenizer_config.json` | Tokenizer (bert-base-multilingual-cased) |
+| `threshold.json` | `{"threshold": 0.95}` |
+| `training_config.yaml` | Full training config + git SHA |
+| `val_metrics.json` | Val set metrics at end of training |
+
+Eval artifacts (in `eval/results/phishing/`):
+| File | Description |
+|---|---|
+| `eval_report.json` | Full test set metrics + confusion matrix + latency |
+
+---
+
+## Changelog
+
+| Version | Date | Notes |
+|---|---|---|
+| v1.0 | 2026-06-13 | Initial release. All eval gates passed. English-only training data. |

v1.0/threshold.json

@@ -0,0 +1,3 @@
+{
+  "threshold": 0.95
+}

v1.0/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

v1.0/training_config.yaml

@@ -0,0 +1,50 @@
+# Phishing detector — v1 baseline config.
+# Decision log (feat/phishing-model, 2026-06-09):
+#   Base model  : bert-base-multilingual-cased — English-capable (§11 decision,
+#                 approved by Insan; diverges from priority's IndoBERT because
+#                 ealvaradob/phishing-dataset is English-heavy).
+#   Legit class : Enron corporate email corpus (emails.csv, 20 K sampled).
+#   Imbalance   : weighted cross-entropy + 3× multiplier on phishing class.
+#
+# Run via:
+#   make data-phishing
+#   make train-phishing config=configs/phishing_v1.yaml
+
+model_type: phishing
+model_name: bert-base-multilingual-cased   # §11 decision: English-capable base
+
+dataset: ealvaradob/phishing-dataset       # primary phishing training data (HF)
+legit_source: emails.csv                   # Enron corpus — negative class source
+
+num_labels: 2   # legit (0) | phishing (1)  — matches PHISHING_LABELS in constants.py
+
+hyperparameters:
+  learning_rate: 2.0e-5         # AdamW (mirrors priority protocol, ML_PIPELINE.md §3)
+  weight_decay: 0.01
+  batch_size: 16
+  gradient_accumulation_steps: 1
+  num_epochs: 4                 # 3–5 with early stopping
+  warmup_ratio: 0.10
+  lr_scheduler_type: linear
+  max_seq_length: 512
+  # Early stopping on RECALL, not macro F1 — false negatives are worst case (§3).
+  early_stopping_metric: recall_phishing
+  early_stopping_patience: 2
+  class_weights: balanced                # inverse-frequency base weights from train dist
+  phishing_class_multiplier: 3.0         # extra multiplier on phishing class weight
+  mixed_precision: bf16                  # falls back to fp16 / none as needed
+
+seed: 42
+
+output_dir: checkpoints/phishing_v1
+
+wandb:
+  project: priormail
+  tags:
+    - model:phishing
+    - stage:baseline
+
+# --- run provenance (auto-recorded) ---
+_git_sha: 94013c0
+_git_dirty: True
+_trained_at: 2026-06-13T10:59:03