inpainting/README.md

---
library_name: diffusers
pipeline_tag: image-to-image
base_model: neonforestmist/Clover-Image-Tiny
license: creativeml-openrail-m
tags:
  - clover-image
  - inpainting
  - stable-diffusion
  - coreml
---

# Clover Image Tiny Inpaint 🍀

An inpainting adaptation of Clover Image Tiny for 512×512 local generation and
on-device Core ML deployment. White mask pixels are regenerated; black pixels
are preserved. The intended target is an SD 1.4-class architecture, rather than
a device-specific model.

The model uses a 9-channel U-Net input:

```text
[noisy latent (4), mask (1), masked-image latent (4)]
```

The base text encoder, VAE, scheduler, safety checker, and tokenizer remain
compatible with Clover Image Tiny. The inpainting export additionally includes
the VAE encoder needed to prepare the masked-image latent on device.

## Diffusers example

```python
from diffusers import AutoPipelineForInpainting, DPMSolverMultistepScheduler
from diffusers.utils import load_image

pipe = AutoPipelineForInpainting.from_pretrained(
    "neonforestmist/Clover-Image-Tiny-Inpaint",
    torch_dtype="auto",
)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
image = pipe(
    prompt="a tiny glass greenhouse glowing in a moonlit garden",
    image=load_image("input.png"),
    mask_image=load_image("mask.png"),
    num_inference_steps=20,
    guidance_scale=6.0,
    padding_mask_crop=96,
).images[0]
image.save("clover-inpaint.png")
```

The mask is a grayscale image: white means “regenerate” and black means
“preserve.” A few useful inpainting prompts are:

- `replace the masked area with a tiny glass greenhouse glowing at night`
- `remove the person from the masked area and continue the background naturally`
- `add a red enamel kettle on the masked countertop`

Use DPM-Solver++, 20 steps, CFG 6.0, and a 96-pixel context margin as the
interactive defaults. Exact-mask compositing preserves every source pixel
outside the edit. The selected v2 checkpoint reduced deterministic held-out
masked MAE from 0.3120 to 0.2517 and improved mean semantic CLIP alignment from
0.2726 to 0.2890 over the previous release.

For repeatable experiments, keep the input image, mask, seed, scheduler, and
step count together. Inpainting is local editing: the unmasked region is
provided as the masked-image conditioning and is also preserved by the native
runtime compositor.

## Core ML and SD 1.4-class deployment

The companion Core ML resource bundle is converted for iOS 18 with a
batch-one U-Net and chunked U-Net resources for the SD 1.4-class 512×512
architecture. The Swift runtime performs classifier-free guidance as two
serial passes to reduce peak memory. The
bundled `VAEEncoder.mlmodelc` creates the masked-image latent locally, so the
input image and mask do not leave the device.

Conversion and the native iOS integration live in the source Clover repo:

- [`coreml-tools/convert_inpaint.sh`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/blob/main/coreml-tools/convert_inpaint.sh)
- [`Clover-iOS`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/tree/main/Clover-iOS)
- [`training/README-INPAINTING.md`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/blob/main/training/README-INPAINTING.md)

## Training provenance

The context-aware v2 recipe distills the pinned official SD 1.5 inpainting
U-Net into Clover's compact nine-channel U-Net while retaining a ground-truth
diffusion loss. Training uses diverse synthetic free-form, multi-region,
object-like, and outpainting masks over the pinned Apache-2.0
`prithivMLmods/Caption3o-Opt` image-caption dataset. A deterministic 128-image
holdout is excluded from optimization and reused for same-mask, same-seed
comparisons against the previous release.

The job runs on Modal under the `guccichungus69` workspace and stores rolling
resumable checkpoints plus the final Diffusers pipeline in the
`clover-image-tiny-inpaint-output` Volume before Core ML conversion. Exact
revisions and objective weights are recorded in `inpainting/config.json` and
the generated `training-summary.json`.

## Citation

If Clover Image Tiny Inpaint is useful in your work, please cite the release:

```bibtex
@software{lozadaperez2026cloverimagetinyinpaint,
  author = {Lukas Lozada Perez},
  title = {Clover Image Tiny Inpaint: Compact SD 1.4-Class Image Inpainting},
  year = {2026},
  url = {https://huggingface.co/neonforestmist/Clover-Image-Tiny-Inpaint}
}
```