File size: 6,731 Bytes

77c266c

---
license: apache-2.0
library_name: diffusers
pipeline_tag: text-to-image
base_model: CompVis/stable-diffusion-v1-4
tags:
  - diffusers
  - remote-sensing
  - optical
  - sar
  - infrared
  - multimodal
  - text-to-image
  - lora
  - stable-diffusion
inference: true
language:
  - en
widget:
  - text: There is a ship in the blue water on the shore.
    output:
      url: demo/opt/demo.png
  - text: There is a ship in the blue water on the shore.
    output:
      url: demo/sar/demo.png
  - text: There is a ship in the blue water on the shore.
    output:
      url: demo/ir/demo.png
---

# BiliSakura/MMDiff-diffusers

Self-contained [Diffusers](https://github.com/huggingface/diffusers) checkpoint for **MMDiff**: text-driven generation of spatially consistent **optical (OPT)**, **synthetic aperture radar (SAR)**, and **infrared (IR)** remote-sensing images from a single prompt.

Converted from [`XinRan-Tang/MM-Diff`](https://huggingface.co/XinRan-Tang/MM-Diff) with the native custom pipeline in [MMDiff-diffusers](https://github.com/Bili-Sakura/MMDiff-diffusers). The optical UNet is a Stable Diffusion v1.4 backbone fine-tuned on optical image–text pairs; SAR and IR style are applied with scene LoRA adapters plus in-memory spatial feature transfer (no disk dumps under `features/`).

Paper: [MMDiff: Multi-modal remote sensing image generation via cross-modality spatial feature transfer](https://doi.org/10.1016/j.isprsjprs.2026.08.018) (ISPRS Journal of Photogrammetry and Remote Sensing, 2026).

## Demo

Prompt: *"There is a ship in the blue water on the shore."* — scene `ship`, 256×256, 50 DDPM steps, `guidance_scale=7.5`, seed `2026`.

| Optical (OPT) | SAR | Infrared (IR) |
| --- | --- | --- |
| ![OPT](demo/opt/demo.png) | ![SAR](demo/sar/demo.png) | ![IR](demo/ir/demo.png) |

## Model details

| Field | Value |
| --- | --- |
| Pipeline class | `MMDiffPipeline` (`pipeline.py`) |
| Backbone | Stable Diffusion v1.4 (`UNet2DConditionModel` + `AutoencoderKL` + CLIP ViT-L/14) |
| Scheduler | `DDPMScheduler`, 1000 training steps, `scaled_linear`, `prediction_type=epsilon` |
| Native resolution | 256×256 |
| Latent channels | 4 (VAE `scaling_factor=0.18215`) |
| SAR / IR adapters | PEFT LoRA under `loras/{sar,ir}/<scene>/` |
| Safety checker | Disabled (remote-sensing imagery) |
| License | Apache-2.0 |
| Training data | [`XinRan-Tang/Optical-SAR-Infrared`](https://huggingface.co/datasets/XinRan-Tang/Optical-SAR-Infrared) |

### Scene LoRAs

`beach`, `bridge`, `desert`, `farmland`, `lake`, `mountain`, `residential`, `river`, `ship`.

SAR and IR images are decoded as single-channel (grayscale) to match the original sampling path. Optical images remain RGB.

## Repo layout

```text
BiliSakura/MMDiff-diffusers/
├── README.md
├── pipeline.py
├── model_index.json
├── demo/
│   ├── opt/demo.png
│   ├── sar/demo.png
│   └── ir/demo.png
├── unet/
├── vae/
├── text_encoder/
├── tokenizer/
├── scheduler/
└── loras/
    ├── sar/<scene>/pytorch_lora_weights.safetensors
    └── ir/<scene>/pytorch_lora_weights.safetensors
```

This folder is a complete Diffusers repo: load the **root**, not a nested variant.

## Load from Hugging Face

```python
import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "BiliSakura/MMDiff-diffusers",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to("cuda")

generator = torch.Generator(device="cpu").manual_seed(2026)
output = pipe(
    "There is a ship in the blue water on the shore.",
    scene="ship",
    height=256,
    width=256,
    num_inference_steps=50,
    guidance_scale=7.5,
    generator=generator,
)
output.opt[0].save("opt.png")
output.sar[0].save("sar.png")
output.ir[0].save("ir.png")
```

## Load from a local clone

```python
from pathlib import Path
import torch
from diffusers import DiffusionPipeline

model_dir = Path("./MMDiff-diffusers").resolve()
pipe = DiffusionPipeline.from_pretrained(
    str(model_dir),
    local_files_only=True,
    custom_pipeline=str(model_dir / "pipeline.py"),
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to("cuda")

generator = torch.Generator(device="cpu").manual_seed(2026)
output = pipe(
    "There is a ship in the blue water on the shore.",
    scene="ship",
    modalities="all",  # or ["opt", "sar", "ir"]
    height=256,
    width=256,
    num_inference_steps=50,
    guidance_scale=7.5,
    generator=generator,
)
output.opt[0].save("demo/opt/demo.png")
output.sar[0].save("demo/sar/demo.png")
output.ir[0].save("demo/ir/demo.png")
```

## Recommended inference settings

| Setting | Value |
| --- | --- |
| Resolution | 256×256 |
| Steps | 50 |
| CFG (`guidance_scale`) | 7.5 |
| `torch_dtype` | `bfloat16` |
| Scheduler | `DDPMScheduler` (shipped) |
| Spatial transfer | attention layers `1..9`, ResNet layer `2`, `resnet_time=1.0` |

`modalities` accepts `"all"` or any subset of `"opt"`, `"sar"`, `"ir"`. SAR/IR generation requires OPT spatial features; the pipeline runs OPT first when they are not supplied via `spatial_features`.

Dependencies: `diffusers`, `transformers`, `accelerate`, `peft`, `safetensors`, `torch`, `pillow`.

## Interface notes

- `output.images` is the first requested modality (Stable Diffusion convention); `output.opt` / `output.sar` / `output.ir` hold per-modality PIL lists.
- `scene` selects packaged SAR/IR LoRAs. Override with `sar_lora_path` / `ir_lora_path` if needed.
- Pass `return_spatial_features=True` to reuse captured OPT features in a later call.

## Intended use

Research on text-driven multi-modal remote-sensing generation, cross-modality spatial transfer, multi-modal fusion, and downstream MMRS data augmentation. Not intended as a general-purpose photorealistic image generator.

## Links

- Paper: [ISPRS Journal of Photogrammetry and Remote Sensing](https://www.sciencedirect.com/science/article/pii/S0924271626004089)
- Homepage: [MMDiff](https://xinr-tang.github.io/MMDiff-homepage/)
- Upstream weights: [`XinRan-Tang/MM-Diff`](https://huggingface.co/XinRan-Tang/MM-Diff)
- Dataset: [`XinRan-Tang/Optical-SAR-Infrared`](https://huggingface.co/datasets/XinRan-Tang/Optical-SAR-Infrared)
- Conversion / pipeline: [MMDiff-diffusers](https://github.com/Bili-Sakura/MMDiff-diffusers)

## Citation

```bibtex
@article{tang2026mmdiff,
  title   = {MMDiff: Multi-modal remote sensing image generation via cross-modality spatial feature transfer},
  author  = {Tang, Haojun and Zhao, Wenda and Cui, Hengshuai and Wang, Haipeng},
  journal = {ISPRS Journal of Photogrammetry and Remote Sensing},
  year    = {2026},
  doi     = {10.1016/j.isprsjprs.2026.08.018}
}
```