---
license: openrail++
base_model:
- stabilityai/stable-diffusion-xl-base-1.0
- h94/IP-Adapter
tags:
- stable-diffusion-xl
- ip-adapter
- controlnet
- garment
- texture-generation
- uv-mapping
---

# Garment UV IP-Adapter (v2)

Fine-tuned IP-Adapter weights that, combined with the [garment UV ControlNet](https://huggingface.co/JorgeAskur/garment-uv-controlnet-v3),
generate **UV-space texture atlases** for 3D garment meshes from a **photo** of a real garment.

This is a delta on top of `h94/IP-Adapter` (`sdxl_models/ip-adapter_sdxl.bin`):
- Image projection layers (4 tensors)
- IP K/V cross-attention projections across all UNet cross-attention layers (140 tensors)
- Total: ~351 M parameters

Trained on (rendered garment view → ground-truth UV atlas) pairs — the views are
the per-sample meshes textured with their target atlases and rendered from 3 angles
(front, +45°, -45°).

## Usage

```python
import torch
from diffusers import (
    AutoencoderKL, ControlNetModel,
    StableDiffusionXLControlNetPipeline, UniPCMultistepScheduler,
)
from PIL import Image
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

# Load the geometry-aware ControlNet
controlnet = ControlNetModel.from_pretrained(
    "JorgeAskur/garment-uv-controlnet-v3", torch_dtype=torch.float16
)
vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet, vae=vae, torch_dtype=torch.float16,
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)

# Load base IP-Adapter, then apply the fine-tuned delta
pipe.load_ip_adapter("h94/IP-Adapter",
                     subfolder="sdxl_models",
                     weight_name="ip-adapter_sdxl.bin")
ft_path = hf_hub_download(repo_id="JorgeAskur/garment-uv-ip-adapter-v2", filename="ip_adapter.safetensors")
pipe.unet.load_state_dict(load_file(ft_path), strict=False)

pipe.set_ip_adapter_scale(0.7)
pipe = pipe.to("cuda")

photo = Image.open("clothing_photo.jpg").convert("RGB").resize((1024, 1024))
normal = Image.open("normal.png").convert("RGB").resize((1024, 1024))

atlas = pipe(
    prompt="garment panels unfolded on canvas, UV texture atlas",
    negative_prompt="3d render, photograph, person wearing, blurry",
    image=normal,
    ip_adapter_image=[photo],
    num_inference_steps=40,
    guidance_scale=5.0,
    controlnet_conditioning_scale=1.0,
    height=1024, width=1024,
).images[0]
atlas.save("atlas.png")
```

## Inputs

- **`image=` (ControlNet):** UV-space tangent normal map, encoded `(N * 0.5 + 0.5) * 255` per channel, black background.
- **`ip_adapter_image=` (this adapter):** photo or render of the garment. Flat-lay product
  shots and renders of the garment on its mesh work best; in-the-wild photos of people
  wearing clothes are out-of-distribution.

## Tuning

- `pipe.set_ip_adapter_scale(0.7)` is a balanced default. Push to `0.9` for stronger
  appearance match (may dominate text guidance), drop to `0.4` to weight the prompt more.
- `controlnet_conditioning_scale=1.0` keeps UV layout strict; lower it if the model is
  forcing the photo into the wrong spatial regions.

## Limitations

- CLIP image encoder is 224×224 — fine repeating textures (knit weave, dense plaid) get
  smoothed in the encoder and may not transfer faithfully.
- Five garment categories from the companion ControlNet (long-shirt, short-shirt,
  long-dress, pants, shorts).

## License

OpenRAIL++ (inherits from SDXL base + IP-Adapter base).