Hugging Face's logo

Spaces:
pavankumarvk
/
Multi_Modal_Deepfake_Detection
Sleeping

App Files Community
Multi_Modal_Deepfake_Detection / PROJECT_SUMMARY.md
pavankumarvk's picture
pavankumarvk
Update PROJECT_SUMMARY.md
fb650c1 verified
preview code
|
Raw
History Blame
14.3 kB

📊 Project Summary & Structure

Complete overview of the Deepfake Detection System

🎯 Project at a Glance

Attribute Details
Name Deepfake Detection System
Version 1.2.0
Purpose AI-powered detection of manipulated images and videos
Framework TensorFlow 2.12 + Gradio
Model EfficientNetV2-B0
Python 3.10.11 (Recommended)
Interface Web-based (Gradio)
License Educational & Research Use

📁 Complete File Structure 

newmultimodal/                      # Root directory
│
├── 📄 README.md                    # Main documentation 
├── 📄 INSTALLATION_GUIDE.md        # Detailed installation instructions
├── 📄 PROJECT_SUMMARY.md           # This file
├── 📄 .gitignore                   # Git ignore rules
├── 📄 .gitattributes               # Git LFS configuration
│
├── 🐍 Python Files
│   ├── app.py                      # Main Gradio application (54 lines)
│   ├── pipeline.py                 # Detection pipeline logic (209 lines)
│   └── rawnet.py                   # Audio model architecture (391 lines)
│
├── 📦 Configuration Files
│   ├── requirements.txt            # Python dependencies (11 packages)
│   ├── packages.txt                # System dependencies (3 items)
│   └── run_app.bat                 # Windows launch script
│
├── 🤖 Model Files
│   ├── efficientnet-b0/            # Image/Video detection model (~87 MB)
│   │   ├── saved_model.pb          # TensorFlow model graph
│   │   ├── keras_metadata.pb       # Keras metadata
│   │   ├── variables/              # Model weights
│   │   │   ├── variables.data-00000-of-00001
│   │   │   └── variables.index
│   │   └── assets/                 # Model assets (if any)
│   │
│   └── RawNet2.pth                 # Audio model weights (~67 MB)
│
├── 🖼️ Example Data
│   ├── images/                     # Test images
│   │   ├── images_lady.jpg         # Real image example
│   │   └── images_fake_image.jpg   # Fake image example
│   │
│   ├── videos/                     # Test videos
│   │   ├── celeb_synthesis.mp4     # Fake video example
│   │   └── real-1.mp4              # Real video example
│   │
│   └── audios/                     # Test audio files (optional)
│       ├── DF_E_2000027.flac
│       ├── DF_E_20000281.flac
│       ├── DF_E_2000031.flac
│       └── DF_E_2000032.flac
│
└── 📂 .git/                        # Git repository (if cloned)

📋 File-by-File Description

Core Application Files

app.py - Main Application

Purpose: Gradio web interface Size: ~1.7 KB Key Features:

  • Two-tab interface (Image, Video)
  • Custom CSS for large UI
  • Example file integration
  • Port configuration

Key Code:

image_interface = gr.Interface(
    pipeline.deepfakes_image_predict,
    gr.Image(height=500),
    gr.Textbox(lines=8)
)
app.launch(share=False, inbrowser=True)

pipeline.py - Detection Pipeline

Purpose: Core detection logic Size: ~6.6 KB Key Components:

  • DetectionPipeline class
  • deepfakes_image_predict() - Image detection
  • deepfakes_video_predict() - Video detection
  • deepfakes_audio_predict() - Audio detection (kept for future)
  • load_audio_model() - RawNet2 loader

Processing Flow:

  1. Load and resize input (224x224)
  2. Normalize pixel values (0-1 range)
  3. Run through EfficientNet model
  4. Get confidence scores
  5. Return classification result

rawnet.py - Audio Model

Purpose: RawNet2 architecture for audio detection Size: ~13.7 KB Note: Optional - kept for future audio feature

Configuration Files

requirements.txt - Python Dependencies 

tensorflow==2.12.0          # Core ML framework
gradio                      # Web interface
facenet_pytorch            # Face detection
numpy                      # Numerical operations
opencv-python              # Image processing
opencv-python-headless     # Headless OpenCV
mtcnn                     # Face detection
moviepy                   # Video processing
librosa                   # Audio processing
torch                     # PyTorch backend
torchvision              # Vision utilities

Total Packages: 11 direct dependencies Installation Time: ~5-10 minutes

packages.txt - System Dependencies 

ffmpeg      # Video encoding/decoding
libsm6      # X11 Session Management library
libxext6    # X11 extensions library

Note: Only required for Linux systems

.gitignore - Version Control

Excludes:

  • Python cache (__pycache__/)
  • Virtual environments
  • IDE files
  • Test/debug scripts
  • Log files

Model Files

EfficientNetV2-B0 Model

Location: efficientnet-b0/ Size: ~87 MB Format: TensorFlow SavedModel Purpose: Image and video deepfake detection

Architecture Details:

  • Input: 224x224x3 RGB images
  • Layers: Efficient compound scaling
  • Output: 2 classes (Real, Fake)
  • Activation: Softmax
  • Optimized for inference speed

Performance:

  • CPU Inference: ~0.5-2 seconds per image
  • Memory Usage: ~500 MB RAM
  • Accuracy: Context-dependent

RawNet2 Model

Location: RawNet2.pth Size: ~67 MB Format: PyTorch state dict Purpose: Audio deepfake detection (optional)

Note: Currently not used in UI but kept for potential future integration

Example Data

Images

File Type Size Description
images_lady.jpg Real ~22 KB Example real image
images_fake_image.jpg Fake ~14 KB Example fake image

Videos

File Type Size Duration Description
celeb_synthesis.mp4 Fake ~204 KB Short Synthesized celebrity video
real-1.mp4 Real ~616 KB Short Real person video

Audio (Optional)

  • 4 FLAC files for audio detection testing
  • Total size: ~205 KB

🔧 Technical Stack

Core Technologies

Technology Version Purpose
Python 3.10.11 Programming language
TensorFlow 2.12.0 Deep learning framework
Gradio Latest Web interface
OpenCV Latest Image/video processing
PyTorch Latest Audio model backend
NumPy Latest Numerical operations

Model Architecture

EfficientNetV2-B0:

  • Compound scaling method
  • MBConv blocks
  • Squeeze-and-excitation
  • Optimized for efficiency

Input Processing:

  1. Resize to 224x224
  2. Convert to RGB
  3. Normalize [0, 1]
  4. Batch processing for videos

Output:

  • Binary classification
  • Confidence percentage
  • Real vs Fake determination

🎯 Key Features

1. Image Detection

  • Input: Single image file
  • Processing: Resize → Normalize → Classify
  • Output: Real/Fake + Confidence %
  • Time: ~1-2 seconds

2. Video Detection

  • Input: Video file (any format)
  • Processing: Frame extraction → Batch analysis → Aggregation
  • Output: Overall Real/Fake + Average confidence
  • Time: ~2-10 seconds (varies by length)
  • Method: Analyzes 5 evenly-spaced frames

3. User Interface

  • Framework: Gradio
  • Layout: Tabbed interface
  • Size: Extra large (1400px width)
  • Components:
    • Large upload areas (500px height)
    • Expanded output boxes (8 lines)
    • Example file integration
    • Drag-and-drop support

📊 Performance Metrics

Speed

  • Image Inference: 0.5-2 seconds
  • Video Inference: 2-10 seconds
  • Model Loading: ~5 seconds (one-time)
  • Startup Time: ~10-15 seconds

Resource Usage

  • RAM: 1-2 GB during inference
  • Disk: ~500 MB total
  • CPU: Moderate usage
  • GPU: Optional (not required)

Accuracy

  • Context-dependent: Varies by content type
  • Best for: Clear facial images, good quality videos
  • Limitations: May struggle with low-quality or heavily compressed media

🚀 Workflow

User Workflow 

1. Clone Repository
   ↓
2. Install Dependencies
   ↓
3. Activate Environment
   ↓
4. Run app.py
   ↓
5. Open Browser (http://127.0.0.1:7860)
   ↓
6. Upload Image/Video or Use Examples
   ↓
7. Click Submit
   ↓
8. View Detection Result

Developer Workflow 

1. Fork Repository
   ↓
2. Clone Locally
   ↓
3. Create Feature Branch
   ↓
4. Make Changes
   ↓
5. Test Thoroughly
   ↓
6. Commit & Push
   ↓
7. Create Pull Request

🔍 Code Organization

app.py Structure 

# Imports
import gradio as gr
import pipeline

# CSS Configuration
custom_css = """..."""

# Interface Definitions
image_interface = gr.Interface(...)
video_interface = gr.Interface(...)

# App Configuration
app = gr.TabbedInterface(...)

# Launch
app.launch(...)

pipeline.py Structure 

# Imports and Setup
import tensorflow as tf
...

# Model Loading
model = tf.keras.models.load_model("efficientnet-b0/", compile=False)

# Pipeline Class
class DetectionPipeline:
    def __init__(self, ...):
        ...
    def __call__(self, filename):
        # Frame extraction and processing
        ...

# Prediction Functions
def deepfakes_image_predict(input_image):
    # Image detection logic
    ...

def deepfakes_video_predict(input_video):
    # Video detection logic
    ...

📚 Documentation Structure

Main Documentation

  1. README.md

  2. PROJECT_SUMMARY.md (This file)

    • Complete overview
    • File descriptions
    • Technical details

🎓 Learning Path

Beginner

  1. Follow installation steps
  2. Run with example files
  3. Understand basic usage

Intermediate

  1. Read full README.md
  2. Understand detection pipeline
  3. Experiment with different files
  4. Modify UI parameters

Advanced

  1. Study pipeline.py code
  2. Understand model architecture
  3. Optimize performance
  4. Contribute enhancements

🔄 Version History

v1.0.0 - Initial Release

  • Image detection
  • Video detection
  • Audio detection
  • Basic UI

v1.1.0 - UI Enhancement

  • Larger interface (1400px)
  • Bigger input areas (500px)
  • Expanded output (8 lines)
  • Better examples integration

v1.2.0 - Cleanup & Documentation

  • Removed audio tab from UI
  • Cleaned project structure
  • Comprehensive documentation
  • Fixed file paths
  • Optimized dependencies

🎯 Future Enhancements

Planned Features

  • Batch image processing
  • Video timeline analysis
  • Heatmap visualization
  • API endpoint
  • Mobile interface
  • Multi-language support
  • Custom model upload
  • Result export (JSON/CSV)

Performance Improvements

  • GPU acceleration
  • Model quantization
  • Caching mechanism
  • Async processing
  • Progress indicators

UI Enhancements

  • Dark/Light theme toggle
  • Comparison view
  • History tracking
  • Confidence visualization
  • Detailed analytics

🤝 Contributing Areas

Area Difficulty Impact
UI Improvements Easy High
Documentation Easy Medium
Bug Fixes Medium High
Performance Hard High
New Models Hard High
API Development Medium Medium

📞 Support Resources

Documentation

  • ✅ README.md - Main guide
  • ✅ QUICKSTART.md - Fast setup
  • ✅ INSTALLATION_GUIDE.md - Detailed install
  • ✅ PROJECT_SUMMARY.md - This overview

External Resources

  • EfficientNet Architecture: Google Research
  • Gradio Framework: Gradio Team
  • TensorFlow: Google Brain Team
  • Open Source Community: For tools and models

⚠️ Important Notes

Do Not Delete

  • efficientnet-b0/ folder - Contains model
  • images/ - Example files for UI
  • videos/ - Example files for UI
  • pipeline.py - Core logic
  • app.py - Main application

Safe to Delete (if needed)

  • audios/ - Not used in current UI
  • RawNet2.pth - Not used in current UI
  • rawnet.py - Not used in current UI
  • cleanup.ps1 - Temporary script

Generated Files (ignored by Git)

  • __pycache__/ - Python cache
  • *.pyc - Compiled Python
  • Test/debug scripts

📊 Project Statistics

Metric Value
Total Lines of Code ~700
Number of Files 15 core files
Documentation Pages 4
Model Size ~154 MB
Example Data ~1 MB
Dependencies 11 packages
Supported Formats 8+ types
Average Inference Time 2-5 seconds

✅ Cleanup Summary

Files Removed

  • ✅ app_fixed.py - Duplicate file
  • ✅ check_tf.py - Debug script
  • ✅ debug_tf.py - Debug script
  • ✅ test_inference.py - Test script
  • ✅ efficientnet-b0.zip - Redundant archive
  • ✅ __pycache__/ - Python cache
  • ✅ pipeline.ipynb - Development notebook

Files Added

  • ✅ .gitignore - Git ignore rules
  • ✅ QUICKSTART.md - Quick start guide
  • ✅ PROJECT_SUMMARY.md - This file

Files Updated

  • ✅ README.md - Complete rewrite
  • ✅ requirements.txt - Added gradio, removed tensorflow-addons
  • ✅ app.py - Enhanced UI, removed audio tab
  • ✅ pipeline.py - Removed tensorflow-addons import

🎯 Project Status

Status: ✅ Production Ready

Checklist

  • Code cleaned and optimized
  • Dependencies resolved
  • Documentation complete
  • Examples working
  • UI enhanced
  • Ready for GitHub
  • Ready for deployment

📖 Quick Reference

Essential Commands 

# Setup
conda create -n deepfake_detector python=3.10.11 -y
conda activate deepfake_detector
pip install -r requirements.txt

# Run
python app.py

# Access
http://127.0.0.1:7860

Essential Files

  • app.py - Start here
  • pipeline.py - Detection logic
  • requirements.txt - Dependencies
  • README.md - Documentation

Essential Directories

  • efficientnet-b0/ - Model
  • images/ - Examples
  • videos/ - Examples

Project is ready for deployment and GitHub publishing! 🚀