README.md

metadata

license: other
tags:
  - deepfake-detection
  - uncertainty-quantification
  - pytorch

Deepfake Triage Plugin — Detector Checkpoints

Four proxy-architecture deepfake detector checkpoints trained for the model-agnostic uncertainty + explainability triage plugin described in the research proposal "A Model-Agnostic Uncertainty and Explainability Plugin for Generalizable Deepfake Detection" (Bhumika Tewari, TBVL Lab, IISER Bhopal). Each checkpoint is loaded by the plugin's FastAPI backend (backend/main.py in Anamitra-Sarkar/deepfake-triage-plugin) and wrapped with MC-Dropout uncertainty estimation, Grad-CAM/attention explainability, and a joint confidence+explanation-quality triage rule.

Update (2026-07-24): retrained on the full corrected dataset. An earlier version of these checkpoints was trained on data affected by a bug in restore_crops_from_hf.py that silently excluded ~750 real videos' worth of already-extracted crops, leaving that run with an artificially severe ~1:35 real:fake ratio instead of FF++'s actual ~1:6. That bug is fixed; the checkpoints and metrics below are from a full retrain on the corrected, complete dataset. The old real-class F1 numbers (xception 0.725, sbi 0.673, vit 0.627, lsda 0.725 at video level) are superseded by the numbers in this card, which are all equal or higher.

Files

File	Architecture (proxy backbone used)	Size
best_xception.pth	Xception proxy: timm xception41 (falls back to EfficientNet-B0 if timm unavailable)	~100 MB
best_sbi.pth	Self-Blended Images (SBI) proxy: EfficientNet-B4	~71 MB
best_vit.pth	UIA-ViT proxy: ViT-Base (vit_base_patch16_224)	~343 MB
best_lsda.pth	LSDA proxy: EfficientNet-B0 + latent-space Gaussian noise injection during training	~16 MB

Important scope note: these are architecturally-diverse proxy backbones standing in for the four architectures named in the research proposal (Xception, SBI, UIA-ViT, LSDA) — they reproduce each paper's general architecture family (CNN / augmentation-based CNN / Vision Transformer / latent-augmented CNN) but not each paper's exact published training recipe (e.g. SBI's self-blending augmentation pipeline, UIA-ViT's patch-consistency loss, or LSDA's specific latent-space augmentation method). Treat these as a working proof-of-concept for the plugin architecture, not a reproduction of the original papers' benchmark numbers.

Training data

Real FaceForensics++ (c23) videos via the xdxd003/ff-c23 Kaggle dataset mirror — folder layout: DeepFakeDetection, Deepfakes, Face2Face, FaceShifter, FaceSwap, NeuralTextures (fake) and original (real).

This run used the full ~7000-video dataset (all 6 fake methods + the full real set), face-cropped via MTCNN and persisted to the Arko007/deepfake-ff-face-crops HF dataset repo across multiple preprocessing sessions (resumable, processed_videos.txt-tracked, no video reprocessed twice). The held-out validation split used for the metrics below has 11,570 frames across 1,049 videos (150 real / 899 fake) — consistent with FF++'s ~1:6 real:fake ratio, confirming the corrected restore actually pulled in the full real class this time.

Training setup (from training/train_ddp.py / the training notebook)

All 4 models: --epochs 20 --patience 5 (early stopping on validation loss), AdamW optimizer, ReduceLROnPlateau scheduler, BCEWithLogitsLoss(pos_weight=n_real/n_fake) for class-imbalance correction, plus a WeightedRandomSampler (per-class weight 1/n_class) during training.

• Xception / SBI / LSDA: --batch_size 64 --lr 1e-4
• ViT: --batch_size 32 --lr 5e-5

Train/val split: 85/15, video-level stratified (not frame-level — see split_samples() in train_ddp.py), so frames from the same video never leak across the split.

Evaluation methodology and results (real, computed — not illustrative)

Computed by training/evaluate_models.py, which reconstructs the exact held-out validation split (seed=42, val_fraction=0.15) and reports accuracy, per-class precision/recall/F1, macro-F1, AUROC, and confusion matrices, at both frame level and video level (video-level = mean probability across a video's frames, since frames from the same video are near-duplicates and accuracy alone is misleading under FF++'s class imbalance).

Caveat, stated plainly: this held-out split was also used during training for checkpoint selection (best validation loss / early stopping). It is not a separate, from-scratch generalization test set. Treat these numbers as trustworthy validation-time performance, not an independent-test-set claim.

Video-level metrics (the numbers that matter for real-world triage)

Model	Accuracy	Real Precision	Real Recall	Real F1	Fake F1	Macro F1	AUROC
xception	0.953	0.770	0.960	0.855	0.972	0.913	0.989
sbi	0.869	0.523	0.973	0.681	0.918	0.799	0.971
vit	0.871	0.529	0.900	0.667	0.920	0.793	0.942
lsda	0.924	0.662	0.953	0.781	0.954	0.868	0.978

Frame-level metrics

Model	Accuracy	Real F1	Fake F1	Macro F1	AUROC
xception	0.926	0.849	0.951	0.900	0.977
sbi	0.836	0.716	0.885	0.800	0.943
vit	0.836	0.692	0.888	0.790	0.913
lsda	0.885	0.776	0.923	0.849	0.955

Reading these honestly: accuracy alone would be misleading here (FF++ is fake-heavy) — that's why real-class F1 and AUROC are the headline numbers. Xception is the strongest all-around (real F1 0.855, AUROC 0.989). SBI and ViT show the largest real-precision vs. real-recall gap (they over-flag real videos as fake more often) but their AUROC (0.94-0.97) shows the underlying probability ranking is still strongly separated — that gap is a threshold-calibration property of those two architectures on this data, not evidence the model failed to learn. LSDA sits in between. No model's F1 collapsed under the class imbalance; the pos_weight + WeightedRandomSampler combination held up.

Full machine-readable results (including confusion matrices) are in eval_results.json in this repo.

Live deployment verification (2026-07-24)

Both the FastAPI backend and the React frontend were deployed to Modal (T4 GPU, CPU fallback if CUDA raises a RuntimeError mid-request) purely to verify the full product end-to-end with these corrected checkpoints — not a permanent hosting solution (the client's proposal asked for the working product, not hosted infrastructure; the Modal deployment was stopped again after verification).

• /health: {"status": "healthy", "cuda_available": true, ...}.
• /detect on a real (non-fake) FF++ validation frame: returned is_fake: false, probability: 0.00069 (correctly confident this is real), weights_source: "trained" (confirms the real checkpoint loaded — not a silently-failed fallback to ImageNet weights), full triage response (entropy/stability/Grad-CAM heatmap) returned correctly.
• Frontend static build served correctly (200, correct title) and was pointed at the Modal backend for this verification pass only.

For day-to-day use, the frontend defaults to sharded per-architecture instances on Render's free tier (CPU-only; see repo frontend/src/App.jsx MODEL_ENDPOINTS).

Uncertainty, explainability, and triage (implementation, not just checkpoints)

See plugin_core/ in the repo:

• uncertainty.py — MCDropoutPlugin (stochastic forward passes → mean probability, variance, entropy), plus calculate_ece / generate_reliability_data for calibration analysis (not yet run against this labeled held-out split — the deployed UI's Calibration tab currently shows illustrative, not measured, calibration-curve numbers; the F1/precision/recall/AUROC numbers above ARE measured).
• explainability.py — Grad-CAM (CNN backbones) / saliency-based attention (ViT), with stability-under-perturbation and spatial-entropy quality metrics.
• triage.py — joint rule combining entropy, explanation stability, and borderline-probability checks into VERIFIED_SAFE / VERIFIED_FAKE / ESCALATE_TO_HUMAN.