--- license: other base_model: Lightricks/LTX-2.5 tags: - ltx-2.5 - ic-lora - video-to-video - cel-animation - lora task_categories: - video-to-video --- # LTX-2.5 22B IC-LoRA — live-action → 2D cel character Converts live-action footage so that **one central character** becomes hand-drawn 2D cel animation while everything else — background, props, bystanders — stays photographic. Trained at **1152×672**, which is the resolution it should be used at. ## Example Live-action reference on the left, generated output on the right. One take, no retouching — the man is fully cel-drawn while the rain-streaked glass and the neon behind him stay photographic. Generated from this prompt: > Live-action footage in a tight close-up on a weathered older man with a grey beard standing > inside a glass phone booth at night, pressing a black telephone handset to his ear as his > face crumples and he weeps with his eyes squeezed shut. Rain streams down the glass behind > him and out-of-focus red and amber neon signs glow in the wet street beyond. --- ## What is in this repo ``` ltx25-iclora-cartoon-cum3250.safetensors the LoRA (rank 32, 312 MB) config.yaml the training config clean/ 11 demo renders, 1152×672, output only side_by_side/ the same 11, 2304×672, reference | output ``` Every clip in `clean/` and `side_by_side/` was generated with the checkpoint in this repo, at 1152×672, seed 42, 30 inference steps, CFG 3.0, STG 1.0, guidance rescale 0.7. --- ## How to use it This is an **IC-LoRA video-to-video** model. Every generation takes **two inputs together**: 1. a **live-action reference clip** — the conditioning video 2. a **text prompt** describing that clip There is **no trigger word**. ### Prompt template > Live-action footage in a **[SHOT TYPE — close-up / medium close-up / medium shot]** of > **[THE ONE CHARACTER THAT WILL BE TURNED INTO 2D — describe them distinctly: age, hair, > clothing, distinguishing features]**, **[WHAT THAT CHARACTER IS DOING]**. > **[SETTING, LIGHTING AND BACKGROUND — everything here STAYS PHOTOGRAPHIC, do not name > other people]**. Only the character in the second bracket is converted. Everything in the last bracket stays live-action — but **anyone you name there will also be converted**, so describe the background as places and objects, not people. Working example: > Live-action footage in a tight close-up on a weathered older man with a grey beard standing > inside a glass phone booth at night, pressing a black telephone handset to his ear as his > face crumples and he weeps with his eyes squeezed shut. Rain streams down the glass behind > him and out-of-focus red and amber neon signs glow in the wet street beyond. ### Rules that matter **Do** name exactly one character; describe them distinctly enough that they cannot be confused with anyone else on screen; open directly with the action or a visual detail; be generous with detail — small objects, textures, secondary movement, background elements. **Don't** name other people in the shot; don't preface with "The video opens with…", "The scene shows…", "We see…"; and **don't try to protect anything with the prompt**. Phrases like "the background stays photographic" or "only the woman is animated" have no effect. What gets converted is learned from the training data, not steered by wording — this was tested directly by rendering the same clip with a prompt naming three people and again naming only one; the output was unchanged. ### Inference settings | | | |---|---| | Resolution | **1152×672**, 97 frames, 24 fps | | Inference steps | 30 | | Video CFG scale | 3.0 | | Video STG scale | 1.0 | | Guidance rescale | 0.7 | | Negative prompt | none | Resolution matters. The style is learned at a specific bucket and does not survive being asked for much more. Render at 1152×672 and upscale afterwards if you need a larger delivery size. ### Running it Rendering goes through the LTX-2 trainer's own validation sampler — no third-party inference code is required: ```bash git clone https://github.com/Lightricks/LTX-2.git ltx2 cd ltx2 && git checkout 400fd31 && uv sync uv run --directory packages/ltx-trainer python scripts/train.py config.yaml \ --disable-progress-bars ``` with `config.yaml` set to `optimization.steps: 1`, `validation.interval: 999`, `validation.skip_initial_validation: false`, and one `validation.samples` entry per clip: ```yaml validation: samples: - prompt: "Live-action footage in a tight close-up on ..." conditions: - type: reference video: /path/to/your_clip.mp4 downscale_factor: 1 temporal_scale_factor: 1 include_in_output: true # writes reference | output side by side video_dims: [1152, 672, 97] inference_steps: 30 video_cfg_scale: 3.0 video_stg_scale: 1.0 guidance_rescale: 0.7 generate_audio: false ``` **Use the DEV transformer, never the distilled one.** The distilled checkpoint produces burnt, overcooked results with this LoRA. Some pipeline examples in the LTX-2 repo default to distilled — override them and assert the resolved path before every launch. --- ## The dataset A **fully synthetic paired-video dataset**, created from scratch. Nothing was filmed or scraped. Cinematic live-action clips were generated, each built around a single clearly identifiable central character. For each clip the first frame was extracted, the central character in that frame was replaced with a 2D cel-animated version of the same character in the same pose and framing, and the clip was then re-animated from that edited frame so the character stays 2D cel throughout the shot while the surroundings remain photographic. That yields both halves of a training pair from the **same shot**: the original clip is the **reference** (conditioning input), the re-animated clip is the **target** (generation goal). Because both derive from one source, framing, camera motion and timing match exactly — the only difference is the character's rendering. **20 pairs / 40 clips** were used for training, conformed to 768×448, 24 fps, 97 frames, silent. **The dataset is public: [`Baberg/ltx-2.5-ic-lora-cartoon-dataset`](https://huggingface.co/datasets/Baberg/ltx-2.5-ic-lora-cartoon-dataset)** — it holds 27 pairs, of which these 20 were used. The other seven were dropped because their targets restyled more than 60% of the frame instead of isolating one character, which taught the model to repaint whole scenes. Removing them, with nothing else changed, cut the share of frame repainted on unseen clips from 33–51% down to 9–24%. See `EXCLUDED.md` in the dataset for the list and the reasoning; the seven also serve as held-out evaluation clips, since this model has never seen them. Captions were generated with **Google Gemini 3.5 Flash** (`gemini-3.5-flash`), captioning the *reference* halves. The caption instruction requires the exact phrase "live-action footage" and requires the one central character to be described distinctly enough to be unambiguous. Two deliberate properties of the conversion: character **reflections are stylised** along with the character, and the targets add a **drawn cast shadow** on the ground — so the model learns the character's effect on the scene, not just a repaint of their silhouette. --- ## How it was trained Two stages. A base run at 768×448, then a resolution adaptation to 1152×672. ### Stage 1 — base, 768×448 | | | |---|---| | Base model | LTX-2.5 22B — **dev** transformer (never distilled) | | Text encoder | `gemma4-12b-with-proj-ltx-2.5-bf16` | | Mode | IC-LoRA video-to-video, `flexible` strategy | | Conditions | `reference` p=1.0, `first_frame` p=0.2 | | Bucket | 768×448×97 @ 24 fps | | Rank / alpha | 32 / 32 | | Learning rate | 2e-4, linear schedule | | Steps | 3000 | | Precision | bf16, gradient checkpointing on | | Hardware | 1× RTX PRO 6000 Blackwell 96 GB | | Wall clock | 4h 22m, peak 57.0 GB | ### Stage 2 — resolution adaptation, 1152×672 The checkpoint in this repo. It **warm-starts from stage 1's step-3000 weights** and continues for 1000 more steps at the higher bucket, so cumulative training is **4000 steps**. The style is already learned at this point; the adaptation only teaches it to hold at more latent tokens. | | | |---|---| | Starting weights | stage 1, step 3000 | | Bucket | 1152×672×97 @ 24 fps (19,656 latent tokens vs 8,736) | | Learning rate | **1e-4** — half the base rate | | Steps | 1000 (cumulative 4000) | | Everything else | unchanged from stage 1 | | Wall clock | 4h 47m, 13.73 s/step, peak **68.25 GB** | The lower learning rate is deliberate. A converged LoRA being moved to a new resolution needs small nudges; restarting at the full 2e-4 risks destroying what already works. **Checkpoint selection.** Checkpoints were saved every 125 steps and swept — each rendered on the same held-out clips with identical settings, then scored on how much of the frame changed relative to the reference. The checkpoints are not equally good: | Adaptation step | Cumulative | Frame repainted | | |---|---|---|---| | 125 | 3125 | 38.5% | over-converts — repaints whole scene | | **250** | **3250** | **14.8%** | **selected — matches the base model's native strength** | | 375 | 3375 | 39.5% | over-converts | | 500 | 3500 | 11.9% | clean, slightly under-converted | | 625 | 3625 | 13.8% | clean | The first ~400 steps are unstable while the LoRA absorbs the resolution change; it settles from step 500 on. **The last checkpoint is not the best one** — step 250 (cumulative 3250) was selected on measured evidence, and is what ships here. --- ## Evaluation Renders are produced side by side (reference | output) and scored by the share of frame whose pixels differ from the reference by more than 28 in any channel — lower is better, *provided the character is still fully converted*. A second figure, the ratio of mean signed difference to mean absolute difference, separates contained conversion (~0.7–0.8) from whole-scene repainting (~0.4). Measured on clips **excluded from training**: | Clip | Frame repainted | Ratio | | |---|---|---|---| | Woman on the phone | 18.2% | **0.81** | cleanest containment measured | | Bartender with a shaker | 14.8% | 0.76 | | | Boy with a chicken | 13.6% | 0.66 | held prop survives | | Dancer mid-turn | 10.6% | 0.60 | | | Phone booth | 33.5% | 0.55 | high because the figure fills the frame | | Subway busker | 21.5% | 0.41 | crowd scene — see limitations | --- ## Limitations **Multi-person scenes convert everyone.** If several people are in frame, all of them are converted, not only the one named in the prompt. This is learned behaviour and prompting cannot override it — tested directly. Use shots with one clear subject. **Crowds pick up outlines.** People moving through the background behind the subject acquire ink outlines rather than staying photographic. **Face size drives facial detail.** Eyes render cleanly — proper lid strokes, defined creases — when the face is large in frame. In wide shots, where the eye spans barely more than one latent cell of the VAE's 32× compressed grid, they degrade to soft, undefined shapes. Close and medium-close framing gives markedly better results. This is a property of the training data's shot distribution, not of the resolution: training at a higher bucket did not improve it. **Aspect ratio matters.** The model is trained at 1.714 (1152×672). Rendering at other aspect ratios reframes the shot. --- ## Credits Base model [LTX-2.5](https://huggingface.co/Lightricks/LTX-2.5) by Lightricks. Trained with the [LTX-2 trainer](https://github.com/Lightricks/LTX-2) at commit `400fd31`.