BiliSakura/MMDiff-diffusers

Self-contained Diffusers checkpoint for MMDiff: text-driven generation of spatially consistent optical (OPT), synthetic aperture radar (SAR), and infrared (IR) remote-sensing images from a single prompt.

Converted from XinRan-Tang/MM-Diff with the native custom pipeline in MMDiff-diffusers. The optical UNet is a Stable Diffusion v1.4 backbone fine-tuned on optical image–text pairs; SAR and IR style are applied with scene LoRA adapters plus in-memory spatial feature transfer (no disk dumps under features/).

Paper: MMDiff: Multi-modal remote sensing image generation via cross-modality spatial feature transfer (ISPRS Journal of Photogrammetry and Remote Sensing, 2026).

Demo

Prompt: "There is a ship in the blue water on the shore." — scene ship, 256×256, 50 DDPM steps, guidance_scale=7.5, seed 2026.

Optical (OPT)	SAR	Infrared (IR)

Model details

Field	Value
Pipeline class	MMDiffPipeline (pipeline.py)
Backbone	Stable Diffusion v1.4 (UNet2DConditionModel + AutoencoderKL + CLIP ViT-L/14)
Scheduler	DDPMScheduler, 1000 training steps, scaled_linear, prediction_type=epsilon
Native resolution	256×256
Latent channels	4 (VAE scaling_factor=0.18215)
SAR / IR adapters	PEFT LoRA under loras/{sar,ir}/<scene>/
Safety checker	Disabled (remote-sensing imagery)
License	Apache-2.0
Training data	XinRan-Tang/Optical-SAR-Infrared

Scene LoRAs

beach, bridge, desert, farmland, lake, mountain, residential, river, ship.

SAR and IR images are decoded as single-channel (grayscale) to match the original sampling path. Optical images remain RGB.

Repo layout

BiliSakura/MMDiff-diffusers/
├── README.md
├── pipeline.py
├── model_index.json
├── demo/
│   ├── opt/demo.png
│   ├── sar/demo.png
│   └── ir/demo.png
├── unet/
├── vae/
├── text_encoder/
├── tokenizer/
├── scheduler/
└── loras/
    ├── sar/<scene>/pytorch_lora_weights.safetensors
    └── ir/<scene>/pytorch_lora_weights.safetensors

This folder is a complete Diffusers repo: load the root, not a nested variant.

Load from Hugging Face

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "BiliSakura/MMDiff-diffusers",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to("cuda")

generator = torch.Generator(device="cpu").manual_seed(2026)
output = pipe(
    "There is a ship in the blue water on the shore.",
    scene="ship",
    height=256,
    width=256,
    num_inference_steps=50,
    guidance_scale=7.5,
    generator=generator,
)
output.opt[0].save("opt.png")
output.sar[0].save("sar.png")
output.ir[0].save("ir.png")

Load from a local clone

from pathlib import Path
import torch
from diffusers import DiffusionPipeline

model_dir = Path("./MMDiff-diffusers").resolve()
pipe = DiffusionPipeline.from_pretrained(
    str(model_dir),
    local_files_only=True,
    custom_pipeline=str(model_dir / "pipeline.py"),
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to("cuda")

generator = torch.Generator(device="cpu").manual_seed(2026)
output = pipe(
    "There is a ship in the blue water on the shore.",
    scene="ship",
    modalities="all",  # or ["opt", "sar", "ir"]
    height=256,
    width=256,
    num_inference_steps=50,
    guidance_scale=7.5,
    generator=generator,
)
output.opt[0].save("demo/opt/demo.png")
output.sar[0].save("demo/sar/demo.png")
output.ir[0].save("demo/ir/demo.png")

Recommended inference settings

Setting	Value
Resolution	256×256
Steps	50
CFG (guidance_scale)	7.5
torch_dtype	bfloat16
Scheduler	DDPMScheduler (shipped)
Spatial transfer	attention layers 1..9, ResNet layer 2, resnet_time=1.0

modalities accepts "all" or any subset of "opt", "sar", "ir". SAR/IR generation requires OPT spatial features; the pipeline runs OPT first when they are not supplied via spatial_features.

Dependencies: diffusers, transformers, accelerate, peft, safetensors, torch, pillow.

Interface notes

• output.images is the first requested modality (Stable Diffusion convention); output.opt / output.sar / output.ir hold per-modality PIL lists.
• scene selects packaged SAR/IR LoRAs. Override with sar_lora_path / ir_lora_path if needed.
• Pass return_spatial_features=True to reuse captured OPT features in a later call.

Intended use

Research on text-driven multi-modal remote-sensing generation, cross-modality spatial transfer, multi-modal fusion, and downstream MMRS data augmentation. Not intended as a general-purpose photorealistic image generator.

Links

• Paper: ISPRS Journal of Photogrammetry and Remote Sensing
• Homepage: MMDiff
• Upstream weights: XinRan-Tang/MM-Diff
• Dataset: XinRan-Tang/Optical-SAR-Infrared
• Conversion / pipeline: MMDiff-diffusers

Citation

@article{tang2026mmdiff,
  title   = {MMDiff: Multi-modal remote sensing image generation via cross-modality spatial feature transfer},
  author  = {Tang, Haojun and Zhao, Wenda and Cui, Hengshuai and Wang, Haipeng},
  journal = {ISPRS Journal of Photogrammetry and Remote Sensing},
  year    = {2026},
  doi     = {10.1016/j.isprsjprs.2026.08.018}
}