---
license: other
license_name: ideogram-non-commercial-model-agreement
license_link: https://huggingface.co/ideogram-ai/ideogram-4-fp8/blob/main/LICENSE.md
tags:
- comfyui
- ai-toolkit
- lora
---

This is an experimental LoRA that adds bbox guided inpainting / editing to the Ideogram 4
model. It is a work in progress, so the files here are snapshots at different
points in time while I adjust training parameters and build a better dataset.

## Prerequisites

### Custom Node

You can find my custom node set on GitHub at [ComfyUI-bitpoet-IG4Inpaint](https://github.com/BitPoet/ComfyUI-bitpoet-IG4Inpaint).
The necessary workflow is included in the node or can be downloaded [here](https://github.com/BitPoet/ComfyUI-bitpoet-IG4Inpaint/blob/main/workflows/ideogram4_reference_workflow.json).

### ComfyUI Changes

Check out or download the [dev-ideogram4-inpaint branch](https://github.com/BitPoet/ComfyUI/tree/dev-ideogram4-inpaint) of my Comfy fork.

## Training

To train with reference images, you currently need to use a slightly adapted fork of AI-Toolkit.
You can find my bitpoet-ideogram4-refimages branch [here on GitHub](https://github.com/BitPoet/ai-toolkit/tree/bitpoet-ideogram4-refimages)

It also includes a fix for the UTF-8 / ANSII error lately popping up on Windows that has jobs fail at startup.

Note that this AI-Toolkit adaption is targeted at Ideogram 4 with reference images and JSON prompts in the dataset editor, so you may not be able to use it to
train regular LoRAs.

I will add a small example dataset at some point.

### Buzzwords (technical details)

What we changed in AI-Toolkit besides the dataset editor:

We added reference-latent token concatenation for Ideogram 4: each clean reference image is VAE-encoded and appended to the packed sequence as
[text | noisy target | clean reference], with its own indicator, MRoPE time coordinate, and clean timestep. The transformer output and
diffusion loss are sliced to target tokens only, while bounding-box JSON prompts provide spatial edit conditioning.

These changes have to be mirrored in ComfyUI as well:

ComfyUI core: Extended the native Ideogram 4 model to accept reference latents and reproduce the training sequence [text | noisy output | clean reference],
including the separate indicator, MRoPE coordinate, clean timestep, and output-only prediction slicing.

Custom node: Ideogram4ReferenceConditioning resizes and VAE-encodes a reference image to match the target latent, then attaches it only to positive
conditioning so the separate unconditional model remains unchanged.