vinukatashidu/landslide-detection-multimodal

Vision-Based Landslide Forecasting Using Satellite Imagery: A Multimodal Late-Fusion Approach

1. Abstract

Landslides are devastating natural hazards, often challenging to predict due to the complex interplay of topographical, geological, and meteorological factors. Traditional computer vision models applied to satellite imagery primarily rely on RGB visual features, which often fail to capture the underlying physical precursors of slope failures. This repository hosts a novel Multimodal Late-Fusion Architecture that integrates pre-event high-resolution optical satellite imagery (Sentinel-2) with physical terrain parameters (SRTM Digital Elevation Models) to forecast landslide susceptibility. By decoupling visual feature extraction from physical terrain analysis, the proposed model achieves a high recall rate of 91%, demonstrating significant potential for proactive disaster warning systems.

2. Methodology & Architecture

The proposed model shifts the paradigm from post-event landslide detection to pre-event landslide forecasting. It processes 6-channel satellite patches representing pre-disaster conditions.

The architecture employs a Late-Fusion Strategy:

• Visual Branch (RGB): A pre-trained ResNet50 backbone (weights frozen in early stages) extracts high-level spatial and textural features from Sentinel-2 B4, B3, B2 bands. It outputs a 2048-dimensional feature vector.
• Physical Branch (Terrain & Vegetation): A custom 4-layer Convolutional Neural Network processes the Sentinel-2 B8 (Near-Infrared), SRTM DEM (Elevation), and derived Slope data. It learns the physical preconditions of landslides and outputs a 256-dimensional feature vector.
• Fusion Head: The 2304 combined features are concatenated and passed through a deep Multi-Layer Perceptron (MLP) with Batch Normalization and Dropout to predict the final landslide vulnerability probability.

3. Experimental Setup & Preventing Data Leakage

A critical flaw in many geospatial machine learning models is spatial data leakage, where adjacent image patches from the same geographic event are randomly split into training and testing sets, artificially inflating performance.

• Geographic Splitting: We utilize a strict GroupShuffleSplit based on distinct landslide_ids. This guarantees that the model is evaluated on completely unseen geographic terrain.
• Differential Learning Rates: During fine-tuning, the pre-trained ResNet50 layers are updated with a highly constrained learning rate ($1 \times 10^{-5}$), while the newly initialized Terrain CNN and Fusion Head learn at a faster rate ($1 \times 10^{-4}$).

4. Results & Performance Metrics

The model was evaluated against a Baseline CNN (RGB only) and a Fine-tuned ResNet50 (RGB only). The Multimodal Late-Fusion approach significantly outperformed both visual-only baselines.

Model	ROC-AUC	F1-Score	Recall	Specificity
Baseline CNN (RGB)	0.6601	0.6276	0.63%	0.61%
ResNet50 Fine-Tuned (RGB)	0.7135	0.6738	70%	0.62%
Proposed Multimodal Fusion	0.7371	0.7100	91%	0.46%

Note: The model is optimized for High Recall (sensitivity) to minimize False Negatives, a crucial requirement for disaster early-warning systems. The optimal decision threshold was analytically determined to be 0.291 via Precision-Recall curve analysis on the validation set.

5. Usage

The model expects a 6-channel tensor of shape (6, 224, 224) normalized to [0,1].

• Channel Mapping: [B4 (Red), B3 (Green), B2 (Blue), B8 (NIR), DEM, Slope]

import torch
from multimodal_model import MultimodalLandslideModel

# Initialize model and load weights from HF Hub
model = MultimodalLandslideModel()
model.load_state_dict(torch.load("multimodal_stage2.pth"))
model.eval()

# Dummy input: rgb (3, 224, 224) normalized via ImageNet, terrain (3, 224, 224)
output = model(rgb_tensor, terrain_tensor)
probability = torch.sigmoid(output)

6. Limitations & Future Work

As an experimental academic prototype, this model operates on static pre-event imagery. It currently lacks integration with real-time meteorological sensor data (e.g., live rainfall metrics, soil moisture) and is geographically constrained to the distribution of the training dataset. Future iterations aim to integrate temporal Vision Transformers and live weather APIs for dynamic, real-time forecasting.