---
language:
  - en
  - id
license: other
tags:
  - text-classification
  - phishing-detection
  - bert
base_model: bert-base-multilingual-cased
pipeline_tag: text-classification
---

# Model Card — PriorMail Phishing Detector v1.0

**Model ID:** `phishing/v1.0`
**Owner:** Faiz (PJK-GM095)
**Date:** 2026-06-13
**Status:** Eval gates passed — pending export approval from Insan + Syafiq

---

## What This Model Does

Binary email classifier: **legit (0)** vs **phishing (1)**.

Used by `prior-mail-backend` to flag potentially malicious emails before they reach the user's inbox. Runs at inference time on every incoming email.

---

## Base Model

`bert-base-multilingual-cased` (Google, 178M params)

Chosen over IndoBERT because the primary training data is English-dominant (~98% EN). mBERT handles English well and retains multilingual capability for future Indonesian phishing data.

---

## Training Data

| Source | Role | Rows (after dedup) |
|---|---|---|
| `ealvaradob/phishing-dataset` subset `texts` | Phishing + some legit | ~20,137 |
| Enron corpus (`emails.csv`) | Legit negative class | ~20,000 sampled |

**Total after deduplication:** ~38,000 rows  
**Train / Val / Test split:** 75 / 12.5 / 12.5 (stratified by label)

Deduplication applied on the first 200 chars of body before splitting to prevent leakage.

**Note:** Dataset is almost entirely English. Indonesian phishing emails are not well-represented. Performance on Indonesian-language phishing may be lower — to be addressed in v1.1 with internal labeled data.

---

## Training Config

See [`configs/phishing_v1.yaml`](../configs/phishing_v1.yaml) for the full reproducible config.

Key hyperparameters:

| Parameter | Value |
|---|---|
| Learning rate | 2e-5 |
| Batch size | 16 per device |
| Max sequence length | 512 |
| Epochs | 4 (early stopped at 3) |
| Warmup ratio | 0.10 |
| Loss | Weighted cross-entropy (phishing_class_multiplier=3.0) |
| Early stopping metric | `recall_phishing` (patience=2) |
| Mixed precision | bf16 |
| Seed | 42 |

---

## Evaluation Results

### Threshold Selection (Validation Set)

Threshold selected on val set to maximize precision while keeping recall ≥ 0.95.

**Threshold: 0.95**

| Metric | Val Set |
|---|---|
| Recall | 0.981 |
| Precision | 0.933 |

### Final Results (Test Set, n=5,722)

| Metric | Result | Gate | Status |
|---|---|---|---|
| Recall | **0.970** | ≥ 0.95 | PASS ✅ |
| Precision | **0.928** | ≥ 0.80 | PASS ✅ |
| Latency p95 (CPU) | **435 ms** | < 500 ms | PASS ✅ |
| F1 | 0.948 | — | — |

### Confusion Matrix (Test Set)

|  | Predicted Legit | Predicted Phishing |
|---|---|---|
| **Actual Legit** | 4,593 (TN) | 79 (FP) |
| **Actual Phishing** | 32 (FN) | 1,018 (TP) |

False negatives (missed phishing): **32 out of 1,050** — 3.0% miss rate.  
False positives (false alarms): **79 out of 4,672** — 1.7% false alarm rate.

---

## Input / Output Format

**Input** (assembled by `src/data/preprocess.build_phishing_input()`):
```
FROM: {sender_email} [SEP] SUBJECT: {subject} [SEP] BODY: {body}
```

If sender or subject are unavailable (e.g. plain-body-only sources), those fields are empty strings. The model still performs well on body-only input — the HF training data was ~98% body-only.

**Output:**
- Raw: softmax probability for class 1 (phishing), range [0, 1]
- Final label: `phishing` if probability ≥ 0.95, else `legit`
- Threshold stored in `threshold.json` alongside the checkpoint

---

## Known Limitations

- **Language:** Trained primarily on English data. Lower recall expected on Indonesian-language phishing. Mitigated by mBERT's multilingual pretraining, but not validated.
- **Sender signal weak:** ~98% of HF phishing training samples had no RFC 2822 headers — sender email is not a reliable feature for this version. Enron legit samples do have sender headers, which may introduce a small asymmetry.
- **Adversarial robustness:** Not evaluated against adversarial phishing (obfuscated URLs, lookalike domains, Unicode tricks). Planned for v1.1.
- **Domain shift:** Trained on public phishing corpora. Real-world phishing targeting Indonesian corporate users may have different patterns.

---

## Artifacts

| File | Description |
|---|---|
| `model.safetensors` | Fine-tuned weights (best checkpoint, step 835 / epoch 1) |
| `config.json` | HuggingFace model config |
| `tokenizer.json`, `tokenizer_config.json` | Tokenizer (bert-base-multilingual-cased) |
| `threshold.json` | `{"threshold": 0.95}` |
| `training_config.yaml` | Full training config + git SHA |
| `val_metrics.json` | Val set metrics at end of training |

Eval artifacts (in `eval/results/phishing/`):
| File | Description |
|---|---|
| `eval_report.json` | Full test set metrics + confusion matrix + latency |

---

## Changelog

| Version | Date | Notes |
|---|---|---|
| v1.0 | 2026-06-13 | Initial release. All eval gates passed. English-only training data. |