license: other
license_name: ideogram-non-commercial-model-agreement
license_link: https://huggingface.co/ideogram-ai/ideogram-4-fp8/blob/main/LICENSE.md
tags:
- comfyui
- ai-toolkit
- lora
This is an experimental LoRA that adds bbox guided inpainting / editing to the Ideogram 4 model. It is a work in progress, so the files here are snapshots at different points in time while I adjust training parameters and build a better dataset.
Prerequisites
Custom Node
You can find my custom node set on GitHub at ComfyUI-bitpoet-IG4Inpaint. The necessary workflow is included in the node or can be downloaded here.
ComfyUI Changes
Check out or download the dev-ideogram4-inpaint branch of my Comfy fork.
Training
To train with reference images, you currently need to use a slightly adapted fork of AI-Toolkit. You can find my bitpoet-ideogram4-refimages branch here on GitHub
It also includes a fix for the UTF-8 / ANSII error lately popping up on Windows that has jobs fail at startup.
Note that this AI-Toolkit adaption is targeted at Ideogram 4 with reference images and JSON prompts in the dataset editor, so you may not be able to use it to train regular LoRAs.
I will add a small example dataset at some point.
Buzzwords (technical details)
What we changed in AI-Toolkit besides the dataset editor:
We added reference-latent token concatenation for Ideogram 4: each clean reference image is VAE-encoded and appended to the packed sequence as [text | noisy target | clean reference], with its own indicator, MRoPE time coordinate, and clean timestep. The transformer output and diffusion loss are sliced to target tokens only, while bounding-box JSON prompts provide spatial edit conditioning.
These changes have to be mirrored in ComfyUI as well:
ComfyUI core: Extended the native Ideogram 4 model to accept reference latents and reproduce the training sequence [text | noisy output | clean reference], including the separate indicator, MRoPE coordinate, clean timestep, and output-only prediction slicing.
Custom node: Ideogram4ReferenceConditioning resizes and VAE-encodes a reference image to match the target latent, then attaches it only to positive conditioning so the separate unconditional model remains unchanged.