anmolshrivastav/distilbert-scam-detector-india

Scam/Spam Message Detector — India (DistilBERT, fine-tuned)

Fine-tuned version of distilbert-base-uncased-finetuned-sst-2-english for binary classification of scam/spam vs. legitimate ("ham") text messages, with a focus on Indian scam patterns (lottery fraud, Aadhaar/KYC phishing, UPI/bank fraud, fake job offers, OTP-theft attempts).

Labels

• 0 / ham: Legitimate message
• 1 / spam: Scam or spam message

Training Data

• Base dataset: scam_hum_india.csv, real-world Indian SMS/message samples (telecom promos, government notices, casual messages), deduplicated.
• Augmented with ~200 additional synthetic examples covering underrepresented scam categories: lottery/prize fraud, Aadhaar/KYC phishing, UPI/bank fraud, fake job offers, and OTP-theft social engineering — added after identifying that the base dataset was skewed toward telecom promotional spam and missed these patterns.
• Class-weighted loss (sklearn compute_class_weight="balanced") used during training to counter class imbalance (~1521 ham vs ~700 spam originally).

Training Procedure

• Base model: distilbert-base-uncased-finetuned-sst-2-english
• Epochs: 3
• Learning rate: 2e-5
• Batch size: 16
• Weighted CrossEntropyLoss via custom Trainer subclass to address class imbalance

Evaluation Results

Metric	Score
Accuracy	0.9971
F1	0.9963
Eval Loss	0.0206

Limitations

• Trained primarily on India-specific scam message patterns (Aadhaar, UPI, telecom, lottery in Indian Rupees); may not generalize well to other regions or currencies.
• English-tokenizer based — code-mixed Hindi/English (Hinglish) text may reduce accuracy.
• Subtle, low-signal scam messages (no amounts, brand names, or urgency cues) are harder to detect and may be misclassified as legitimate.
• A portion of spam examples are synthetically generated to patch category gaps; performance on real-world messages of these types should be validated further before production use.

Usage

from transformers import pipeline
clf = pipeline("text-classification", model="anmolshrivastav/distilbert-scam-detector-india")
clf("Congratulations! You've won 50000 rupees, click link to claim")

Quantized ONNX Version

A dynamically quantized (INT8) ONNX version model.onnx is available in the onnx/ subfolder for faster CPU inference with a smaller footprint. Also a onnx/model_fp32.onnx (unquantized ONNX version)

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

model = ORTModelForSequenceClassification.from_pretrained(
    "anmolshrivastav/distilbert-scam-detector-india", subfolder="onnx"
)
tokenizer = AutoTokenizer.from_pretrained(
    "anmolshrivastav/distilbert-scam-detector-india", subfolder="onnx"
)

clf = pipeline("text-classification", model=model, tokenizer=tokenizer)
clf("Congratulations! You've won 50000 rupees, click link to claim")