Card rewrite: audio-only framing, examples, honest eval status

.gitattributes

@@ -33,3 +33,11 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/example_1960s_era.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/example_1960s_era.png filter=lfs diff=lfs merge=lfs -text
+assets/example_dance_tempo.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/example_dance_tempo.png filter=lfs diff=lfs merge=lfs -text
+assets/example_hiphop_identity.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/example_hiphop_identity.png filter=lfs diff=lfs merge=lfs -text
+assets/example_pitch_whisper.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/example_pitch_whisper.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -21,68 +21,73 @@ tags:
 - proof-of-concept
 ---
 
-# LTX-2.3 22B "Helium": an audio-only IC-LoRA (experimental, very much a proof of concept)
+# LTX-2.3 22B "Helium": an audio-only IC-LoRA (experimental, proof of concept)
 
-**edit**: after some quick evals, it looks like passing in an audio file does indeed alter the output, with pitch tones even changing the identity of the speaker in t2v. even speaker or songs influence, as does audio from the 1960s, which changes the video to match the era. 
+Last updated: 2026-05-31
 
-initial findings: **set strength to no higher than 0.5 to start - anything above that can be highly unstable and toward 1.0 will result in garbled video**
+An IC-LoRA for LTX-2.3-22B (distilled) where the in-context reference is **only audio**: no image, no video, no init frame. The model still generates audio and video jointly, but the only thing you hand it as a reference is an audio file. I trained it on a narrow, measurable task (a voiced tone is meant to steer the pitch of the generated speech, the "helium" idea), and in practice the audio reference visibly steers the whole output: a voice or a music track shifts the speaker's identity, the setting, even the era. That broader transfer is the interesting part. The pitch task was just a clean probe for whether an audio-only reference could do anything at all.
 
-This is an IC-LoRA for LTX-2.3-22B (distilled) where the in-context reference is *only audio*, no image and no video. I trained it with a fork of the LTX-2 trainer ([fblissjr/LTX-2 @ audio-guidance-iclora-vtv](https://github.com/fblissjr/LTX-2/tree/audio-guidance-iclora-vtv), the `audio_reference` strategy) plus a pile of ComfyUI nodes and notes ([fblissjr/ComfyUI-AudioLoopHelper](https://github.com/fblissjr/ComfyUI-AudioLoopHelper/tree/main/docs/audio_iclora)). And yes, the repo's scope has wandered way past its name; that's a recurring theme for me.
+> **Status:** evaluated a few different ways, and it clearly produces interesting effects, the audio reference steers speaker identity, scene, and era. It has *not* been through a rigorous, controlled test, and the specific pitch-tracking it was trained for is not cleanly confirmed. Treat the examples below as illustrative single takes, not benchmarks.
 
-Fair warning up front: this is a POC, not something useful beyond setting a pattern for how to train (or not train) an audio IC-LoRA. The task it learns is pointless on purpose. If you came here for a working audio-driven LoRA, this isn't it. It's one experiment of possibly more to follow, and a test rig to answer one question: can an audio-only IC-LoRA be trained and actually work?
+## Model Files
 
-Every IC-LoRA I could find conditions on an image or a video or a video+audio (Cameraman, depth/pose/canny, LipDub's face + driving audio, and so on). I couldn't find one whose reference is *just audio*, controlling something the base model doesn't already do on its own. So can you even train that, given LTX-2.3 is jointly trained on both video and audio? What happens if you modify the trainer so the *reference* is audio-only (no image or video guide), even though the model still generates both? Not quite sure, but I wanted to find out.
+- `lora_weights_step_02000.safetensors`: the final checkpoint (2000 steps).
+- `lora_weights_step_01500.safetensors`: an earlier checkpoint, about 8% apart by weight norm, included to compare. Both are clean (960 finite bf16 tensors each).
+- [`ltx2_audio_reference.yaml`](ltx2_audio_reference.yaml): the training config.
+- [`data_recipe.md`](data_recipe.md): how the dataset was built.
+- Training fork: [fblissjr/LTX-2 @ audio-guidance-iclora-vtv](https://github.com/fblissjr/LTX-2/tree/audio-guidance-iclora-vtv) (the `audio_reference` strategy).
 
-To test whether the audio reference is actually doing the work, I needed a task that's measurable, that the base model doesn't already do easily, and that *only* the audio reference can carry, so caption and init frame can't leak it. Pitch fits ("helium"). It would've been a stronger test still if the pitch wandered randomly through each clip, since then nothing but the reference could say what to do moment to moment, but that's a next-run idea (more in the data recipe).
+## Model Details
 
-The reference is a voiced tone at some pitch; the target is a real talking-head clip whose speech is pitch-shifted to match that tone. The model's job is to read the pitch off the tone and make the generated speech match it. The test is simple: feed it a tone, measure the output's pitch, and check that it tracks, and that it doesn't just do that anyway with no LoRA. Pitch is useless as a feature (you can pitch-shift audio with DSP for free), which also makes it cheap to build a dataset around and a clean thing to measure.
+- **Base Model:** LTX-2.3-22B distilled (`ltx-2.3-22b-distilled-1.1`), a *joint* audio-video model (which matters for the caveats).
+- **Training Type:** IC-LoRA, `audio_reference` strategy.
+- **Control Type:** **Audio.** One in-context audio reference, no image and no video. The model generates both modalities; only the reference is audio-only.
+- **Reference Downscale Factor:** not applicable to audio (defaults to 1; the loader's "couldn't find reference_downscale_factor" warning is harmless).
+- **Adapted modules:** the audio stack only, `audio_attn1`/`audio_attn2` (q/k/v/out) and `audio_ff`, across all 48 blocks. Nothing on the video or cross-modal side.
+- **Rank / alpha:** 32 / 32, no dropout. **Reference:** appended clean at negative RoPE positions (out-of-timeline context, the ID-LoRA convention).
+- **Training:** 2000 steps, batch 1, lr 2e-4 (adamw8bit), bf16, int8-quanto plus block-swap to fit a single 24 GB 4090, gradient checkpointing. Trained at **256x256 with 2-second audio references**, a deliberate limitation set by consumer-grade hardware.
+- **Format:** plain LoRA safetensors, diffusers/PEFT convention (`lora_A`/`lora_B` under `diffusion_model.`, no alpha keys), bf16. ComfyUI's LTX IC-LoRA loader reads it directly, no conversion.
 
-It can be trained architecturally, but the eval is still pending on whether it actually does the thing it's supposed to. And if it doesn't, that could be the data recipe, the training code, any number of things. There are a lot of ways to get an IC-LoRA wrong, and the big one is this: you can't let the signal you want the audio to carry leak in from anywhere else. Where this could go if it works: voice/accent/emotion transfer, audio-driven style. And if it doesn't, better to find that out before sinking significantly more time into it than I already have.
+## Using in ComfyUI
 
-Sharing this prior to evals since I'm still writing the code to do the inference, and also to get it out there and see if others can take it further (or see what I may have gotten wrong and do it better). Really just want to see what's possible.
+1. Copy the LoRA into `models/loras`.
+2. Install the custom nodes from **[ComfyUI-AudioLoopHelper](https://github.com/fblissjr/ComfyUI-AudioLoopHelper)**. The audio IC-LoRA loader and the **Add Audio IC-LoRA Guide** node are not in stock ComfyUI-LTXVideo, so you need this repo.
+3. Use the example workflow [`audio-ic-lora_single-pass.json`](https://github.com/fblissjr/ComfyUI-AudioLoopHelper/blob/main/example_workflows/audio-ic-lora_single-pass.json): load the LoRA with the audio IC-LoRA loader, feed your audio through the Add Audio IC-LoRA Guide node with a neutral caption, and generate audio and video together.
 
-## model details
+How it works and the eval notes are in [`docs/audio_iclora`](https://github.com/fblissjr/ComfyUI-AudioLoopHelper/tree/main/docs/audio_iclora).
 
-- **Base model:** LTX-2.3-22B distilled (`ltx-2.3-22b-distilled-1.1`), a joint audio-video model (which matters for the caveats below).
-- **Type:** audio-only IC-LoRA, trained with the `audio_reference` strategy.
-- **Control:** one in-context audio reference (a voiced tone), meant to steer the pitch of the generated speech.
-- **Adapted modules:** the audio stack only, `audio_attn1`/`audio_attn2` (q/k/v/out) and `audio_ff`, across all 48 blocks, nothing on the video or cross-modal side. The bet is that pitch transfer is audio→audio; it's a bet, not a proven thing.
-- **Rank / alpha:** 32 / 32, no dropout.
-- **Reference conditioning:** appended clean at negative RoPE positions, strength 1.0, as out-of-timeline context (the ID-LoRA convention), not something the model generates.
-- **Training:** 2000 steps, batch 1, lr 2e-4 (adamw8bit), bf16, int8-quanto plus block-swap so it fits a single 24 GB 4090, gradient checkpointing.
-- **Format:** plain LoRA safetensors in the diffusers/PEFT convention (`lora_A`/`lora_B` under `diffusion_model.`, no alpha keys, no metadata), bf16. Not a comfy-specific format; ComfyUI's LTX IC-LoRA loader reads it directly (the same one the official Lightricks IC-LoRAs use), no conversion.
-- **Reference Downscale Factor:** not applicable to audio. The loader's "couldn't find reference_downscale_factor" warning is harmless and defaults to 1.0.
-- **Status:** trains end to end and fits a 24 GB 4090 (but training on distilled); pitch-tracking efficacy is still under evaluation.
+### Usage recommendations
 
-## what's in here
+- **Strength: start at ~0.5.** Higher increases instability, and toward 1.0 the video garbles. The usable ceiling depends on the reference and the generation, so if it breaks up, back off. 0.3 to 0.5 is the working band.
+- **Caption:** keep it neutral and attribute-free, so the audio reference (not the caption) drives the attribute.
+- **Reference:** the reference is audio only. Any audio works; the effect lands as a global attribute (voice, genre, era, scene).
 
-Two LoRA checkpoints, at 1500 and 2000 steps (2000 is the final one; 1500 is included so you can compare). They're about 8% apart by weight norm, so 1500 isn't a stale copy of 2000, and both come out clean: 960 finite bf16 tensors each. Plus links to the [training code fork for audio-only IC-LoRA](https://github.com/fblissjr/LTX-2/tree/audio-guidance-iclora-vtv), [training config](ltx2_audio_reference.yaml) and a [data recipe](data_recipe.md).
+### Examples
 
-## the whole data part matters, a lot
+Single takes: same model, a neutral caption, and the only reference fed in is an audio file. Click a thumbnail for the video.
 
-I come from a data background, and even with that, training an IC-LoRA with purely audio is hard and almost needs a visual whiteboard to make sure you get it right. If the reference and target share content, the caption ends up controlling the attribute and the reference just goes along for the ride. For the *audio* to be in control, the reference and target have to differ in content and share only the attribute. So the reference is a bare tone (pitch, nothing else), the target is real speech (different content) shifted to that pitch, and the caption is a constant, pitch-free string. "Matched content" and "the audio controls it" can't both be true at once, which is the part that took me a while to see. What happens when you do that without the video it was trained on I'm unsure of still until evals come in.
+| result | audio reference | what happens (prompt shown) |
+|---|---|---|
+| <a href="assets/example_pitch_whisper.mp4"><img src="assets/example_pitch_whisper.png" width="200"></a> | a voiced tone | the trained "helium" task: the generated speech drops to a whisper. `A person says, "I made an audio IC Lora"` |
+| <a href="assets/example_hiphop_identity.mp4"><img src="assets/example_hiphop_identity.png" width="200"></a> | hip-hop audio | the speaker and setting shift to match. `A person says, "Hello, how are you doing today?"` |
+| <a href="assets/example_dance_tempo.mp4"><img src="assets/example_dance_tempo.png" width="200"></a> | a high-tempo dance track | the delivery speeds up. `A person says, "I made an audio IC Lora"` |
+| <a href="assets/example_1960s_era.mp4"><img src="assets/example_1960s_era.png" width="200"></a> | a 1960s broadcast | the video takes on a vintage 1960s look. `A person says calmly, "Bandoco is good"` |
 
-## the data, and the one hole in it
+## Dataset
 
-Built from real talking-head footage (the [CelebVHQ](https://huggingface.co/datasets/noakraicer/ID-LoRA-CelebVHQ) and [TalkVid](https://huggingface.co/datasets/noakraicer/ID-LoRA-TalkVid) preprocessed sets). For each clip I measure its natural speaking pitch, pick a target pitch a few semitones off (capped at ±7 so the pitch-shift doesn't sound broken), shift the speech to that pitch while keeping the timing so the lips still line up, and render at 256×256 (the limits of consumer-grade hardware).
+For the audio reference to be in control (not the caption), the reference and target must differ in content and share only the attribute. So the reference is a bare voiced tone, the target is real talking-head speech shifted to that pitch, and the caption is a constant pitch-free string. Built from real footage ([CelebVHQ](https://huggingface.co/datasets/noakraicer/ID-LoRA-CelebVHQ) + [TalkVid](https://huggingface.co/datasets/noakraicer/ID-LoRA-TalkVid)): measure each clip's natural pitch, shift its speech a few semitones (capped at ±7) while keeping timing so lips still line up, render at 256x256, and synthesize a 2-second tone at that pitch with timbre varied independently so the model cannot read timbre instead. Caption always `a person speaking`. Kept a pair only if the shifted pitch re-measured within 30 Hz of target: 292 clean pairs, 76 to 353 Hz.
 
-The reference is a synthesized voiced tone at that same pitch, with the timbre varied independently of pitch so the model can't cheat by reading timbre instead. The caption is always "a person speaking", gender-neutral on purpose, since gender tracks pitch and a gendered caption would leak the thing the tone is supposed to own. No init frame, so the tone is the only non-caption input. Finally I keep a pair only if the shifted pitch re-measures within 30 Hz of target, which left 292 clean pairs out of 300, spanning 76 to 353 Hz.
+The hole: the target pitch is a bounded shift off each clip's *own* pitch, so the two correlate about 0.75, and the speaker's face also hints at their natural pitch. So the tone is not the *only* route to the answer. That does not wreck a positive result (sweep the tone with everything else fixed and watch the pitch follow), but it makes a *null* ambiguous. The fix is a decorrelated rerun (random target pitch, formant-preserving shift); that is the next dataset.
 
-Where I think I messed up: because the target pitch is a bounded shift off the clip's *own* pitch, the two correlate at about 0.75. And the video is the original speaker, whose face says a fair amount about their natural pitch. So the tone isn't the *only* thing that knows the answer; the speaker's appearance is a second route. That doesn't wreck a positive result (sweep the tone with everything else held fixed, and if the pitch follows, the tone is doing it), but it does make a *null* result ambiguous: you can't tell "ignored the reference" from "could've guessed it from the clip". The fix is to decorrelate, with a random target pitch and a formant-preserving shift, and that's the next dataset.
+## Caveats
 
-## running it in comfyui
+- The pitch-tracking eval is **not a clean win**. A controlled reference-swap (fixed prompt and seed, sweep the tone) came back noisy: the base arm leaks the reference tone into the measured audio and the LoRA arm often comes out unvoiced, so the with-vs-without pitch slope is not trustworthy yet. The transfer effect in the examples is obvious; the narrow pitch number is not.
+- The base model is trained on audio and video **jointly**. By holding video back (audio-only reference, audio-only adaptation) I am deliberately isolating the audio path, and it is genuinely possible that is the wrong cut and pitch needs the video in the loop. A null would not prove audio-only is impossible; the next lever is the cross-modal bridge modules (`audio_to_video_attn`, `video_to_audio_attn`) the identity-transfer ID-LoRA uses.
+- Where this could go if pursued: voice / accent / emotion transfer, audio-driven style and scene. This was just a POC to see if it works. It seems to?
 
-The ComfyUI side (the IC-LoRA loader, the node graph for feeding a tone in as the audio reference, and the F0-tracking eval workflow) all lives in the [ComfyUI-AudioLoopHelper repo](https://github.com/fblissjr/ComfyUI-AudioLoopHelper) with some WIP inference nodes for audio-only IC-LoRA guides, so I won't duplicate it here. The short version: load the IC-LoRA, feed the audio file / voiced tone as the audio reference with a constant pitch-free caption, generate audio and video together, then measure the output's pitch against the tone. One detail that matters here: encode the reference tone through the audio VAE in fp32, the same precision the training data was encoded at. The LoRA itself is bf16 and that's fine; it's specifically the audio-VAE encode of the reference that should be fp32, since a lower-precision encode nudges the reference latent off what the model trained on. Full setup and the WIP eval graph are in the repo.
+## Citation
 
-## does it actually work?
-
-I don't know yet, I still need to get inference code working that I can trust. What I can say for certain is that it *trains*: it fits a 4090, there are no NaNs, the checkpoint looks healthy, and the reference measurably moves the training loss, so the model is at least paying attention to *something* in it. Whether that turns into clean pitch-tracking at generation time is still being evaluated; the inference path was still being debugged when I put this up. So treat any "it tracks pitch!" claim as unproven until there's an actual with-IC-LoRA-vs-without pitch slope to back it up.
-
-The bigger caveat: this base model was trained on audio and video jointly. By holding the video back (audio-only reference, audio-only adaptation), I'm deliberately isolating the audio path, and it's genuinely possible that's the wrong cut. Maybe pitch just isn't steerable from audio alone in a model where the two are this entangled, and you'd need the video in the loop, and that explains why nobody is doing it and LTX didn't build a training path for it. A null result wouldn't prove audio-only is impossible; it'd mean isolate-and-test came back "no", and the next thing to try is coupling the video back in. Which is fine; finding that out was the whole point. If the audio-only adapter turns out too weak, the obvious next lever is adding the cross-modal bridge modules (`audio_to_video_attn`, `video_to_audio_attn`) that the identity-transfer ID-LoRA uses.
-
-## credits and lineage
-
-The trick of putting the audio reference at negative RoPE positions comes from Lightricks' LipDub and the ID-LoRA work (cited below); the pitch-gate framing and the tone-as-tracer idea are mine. And to be upfront: my read on *how* the conditioning behaves is mostly from poking at it during inference in ComfyUI, not a clean ablation, so take the mechanism explanations as informed guesses rather than gospel.
+The trick of putting the audio reference at negative RoPE positions comes from Lightricks' LipDub and the ID-LoRA work below. My read on *how* the conditioning behaves is from poking at it in ComfyUI, not a clean ablation, so take the mechanism explanations as informed guesses.
 
 ```bibtex
 @misc{dahan2026idlora,
@@ -95,6 +100,10 @@ The trick of putting the audio reference at negative RoPE positions comes from L
 }
 ```
 
-Thanks to WepeNerd ([HF](https://huggingface.co/WepeNerd)) for the whole idea of an audio-only IC-LoRA and a ton of help thinking through how an IC-LoRA trains and learns and coming up with the idea of 'helium' (the hard part); to the LTX-2 community trainer, plus Musubi Tuner and Kijai's work, for the infra and for getting me looking at the audio side in the first place; and to Throttlekitty for the DM that pulled me into looking at LTX-2 to begin with.
+## Acknowledgments
+
+Thanks to WepeNerd ([HF](https://huggingface.co/WepeNerd)) for the whole idea of an audio-only IC-LoRA and for thinking through how an IC-LoRA learns (including "helium"); to the LTX-2 community trainer, Musubi Tuner, and Kijai's work for the infra; and to Throttlekitty for pulling me into LTX-2 in the first place.
+
+## License
 
-Base model is by Lightricks, and the LTX-2 community license applies. See https://github.com/Lightricks/LTX-2/blob/main/LICENSE for the full terms.
+Base model is by Lightricks under the **LTX-2 community license**. See https://github.com/Lightricks/LTX-2/blob/main/LICENSE for the full terms.

assets/example_1960s_era.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:164ceb9cf7b57e29e03d808661b3f33c239837acc60d8f196fa9c90e40f317ea
+size 923676

assets/example_dance_tempo.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6f0d0d0112a8994d86c116dea0313b5b0c13678fe19cb60aad730673e41449c
+size 1543748

assets/example_hiphop_identity.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d1fc113e3083cdae153efe911ab07d6c028bef0a607933b0890dbd7ae244e41
+size 1263677

assets/example_pitch_whisper.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3ddededb0aad7dc9d9aa09395d5d64e2ee0df769f0d3f774910199f9377d7f1
+size 1132721

data_recipe.md

@@ -1,18 +1,18 @@
-# how the "helium" dataset was built
+# How the "Helium" Dataset Was Built
 
 This is the companion to the model card; the actual recipe used here and how to rebuild it. The *why* (why pitch, why a bare tone, the seesaw, the correlation hole) is all in the README; this is just the how. It's not perfect, but hopefully provides a way for others to make better ones.
 
-## where it comes from
+## Where It Comes From
 
 Real talking-head footage, so the target side has no synthetic-to-real gap: the [CelebVHQ](https://huggingface.co/datasets/noakraicer/ID-LoRA-CelebVHQ) and [TalkVid](https://huggingface.co/datasets/noakraicer/ID-LoRA-TalkVid) preprocessed sets. Both are 512×512, 25 fps, ~5-second (121-frame) clips; I re-render the targets at 256×256 due to consumer grade GPU limitations (and video wasn't the target goal).
 
-## how each pair is made
+## How Each Pair Is Made
 
-For a clip, I measure its natural speaking pitch, then pick a target pitch that's a bounded shift off it (at most +-7 semitones, ~3.6 on average), and pitch shift the speech to that target while keeping the timing so the lips still line up. The video stays untouched. The reference paired with it is a synthesized voiced tone at the same target pitch: 2 seconds, 16 kHz, mono, with the timbre varied across four flavors that are decorrelated from pitch so the model can't read timbre as a pitch shortcut. Every pair gets the same caption, "a person speaking", and no init frame, so the tone is the only non-caption input.
+For a clip, I measure its natural speaking pitch, then pick a target pitch that's a bounded shift off it (at most ±7 semitones, ~3.6 on average), and pitch shift the speech to that target while keeping the timing so the lips still line up. The video stays untouched. The reference paired with it is a synthesized voiced tone at the same target pitch: 2 seconds, 16 kHz, mono, with the timbre varied across four flavors that are decorrelated from pitch so the model can't read timbre as a pitch shortcut. Every pair gets the same caption, "a person speaking", and no init frame, so the tone is the only non-caption input.
 
 Then a quality gate: keep a pair only if the shifted speech re-measures within 30 Hz of its target pitch. That drops the clips where the pitch-shift fell apart and leaves **292 clean pairs out of 300**, spanning 76 to 353 Hz. Two correlations worth knowing: reference timbre vs. pitch comes out near zero (~0.03), which is what I wanted; target pitch vs. the clip's *own* natural pitch is ~0.75, because the target is a bounded shift off it, which is the known weak spot (the README explains why it matters). The fix for the next run is to pick the target pitch at random, independent of the clip, with a formant-preserving shift.
 
-## what the dataset looks like on disk
+## What the Dataset Looks Like on Disk
 
 This is the layout of the *dataset itself*, what the trainer reads, not of this HF download. The dataset isn't shipped here (it's derived from the CelebVHQ/TalkVid sets above, and it's large); this release contains only the IC-LoRA checkpoints, the training config, and this recipe (and a link to the github trainer fork). The structure below is what you'd produce by rebuilding it.
 
@@ -32,7 +32,7 @@ pitch_ref_gate_v1/
 
 Each row in `manifest_train.jsonl` carries the video and reference paths, the caption, the natural / target / actual (re-measured) pitches, the shift in semitones, a timbre id, and a train/heldout split label. One heads-up if you're reproducing: the trainer finds samples by globbing the precomputed `.pt` files and doesn't read that split label, so it trains on everything; the label is informational unless you wire it up yourself.
 
-## rebuilding it
+## Rebuilding It
 
 The scripts and the `audio_reference` strategy live in the trainer fork, [fblissjr/LTX-2 @ audio-guidance-iclora-vtv](https://github.com/fblissjr/LTX-2/tree/audio-guidance-iclora-vtv), under `packages/ltx-trainer/`. Clone it and set up the env (it uses `uv`), then run these from `packages/ltx-trainer/` (paths below are relative to it):
 
@@ -56,14 +56,11 @@ uv run python scripts/train.py ltx2_audio_reference.yaml
 
 The reference encode runs the audio VAE in float32 to match the target-audio encode; the LoRA itself is bf16. Target modules, rank, and the rest of the setup are in `ltx2_audio_reference.yaml` and the README.
 
-## what I'd change next time
+## What I'd Change Next Time
 
-A few, and the organizing idea first: the mechanism here, a clean reference at negative RoPE positions, is built for attributes that are *time-invariant* (an average pitch, a timbre, a voice). That's my read of it, not something I've tested, but it shapes what's worth trying next.
+The organizing idea first: a clean reference at negative RoPE positions suits *time-invariant* attributes (an average pitch, a timbre, a voice). That is my read, not something I tested, but it shapes the list.
 
-The 0.75 correlation is the thing to fix, and I'd do it by construction rather than by luck: reuse each clip at several different target pitches, each with its own matching reference tone, caption held constant. Then the same face shows up at a bunch of pitches, so identity can't predict the answer and the reference tone is the only thing that varies with it, and as a bonus it multiplies a dataset that's way too small. That's the first thing I'd run; it isolates the core question (does the reference steer pitch at all) with no shortcut. A simpler partial version is just to pick the target pitch at random, independent of the clip's natural pitch, with a formant-preserving shift so a deep voice pushed high doesn't chipmunk.
-
-The natural next direction, and the actual point of all this, is voice transfer: a reference of someone speaking, and the generated speech adopts that voice. Voice identity is time-invariant, so it fits this mechanism better than anything time-varying would. It's harder than pitch, though, for the same reason the 0.75 hole exists: in a joint audio-video model the generated face carries a voice prior, and a pure-audio reference has to override it (one scalar for pitch, a whole voice for identity). The open question is whether you need the face at all, and the clean test is to train two variants on the same voice data, one with an audio-only reference and one with image+audio, and compare. The official ID-LoRA uses image+audio for this, which is weak evidence that coupling helps; pure-audio is the more interesting bet that might not land. The pitch gate is the leading indicator: if pure-audio pitch doesn't move, pure-audio voice almost certainly won't.
-
-What I would not chase, and I'd said the opposite earlier, is a pitch that varies over time, a contour the speech has to follow. It's tempting because a contour can't be guessed from a face, but it's probably the wrong tool for this port. A contour only means something if the model lines up "reference moment τ" with "output moment τ", and the negative-RoPE reference sits in a disjoint region with no alignment to the target's timeline, so there's nothing to learn that mapping from. If you want time-varying control (prosody, intonation, rhythm), the right mechanism is an *aligned* driving channel at positive positions, the way LipDub's audio drives lips frame by frame. That's a different experiment, not this one.
-
-And if the data is fine but the audio-only adapter just turns out too weak, the lever is the model side: add the cross-modal bridges (`audio_to_video_attn`, `video_to_audio_attn`, see the README), not more data.
+- **Fix the 0.75 by construction, not luck.** Reuse each clip at several target pitches, each with its own reference tone, caption held constant. The same face then appears across many pitches, so identity can't predict the answer and the tone is the only thing that co-varies, and it multiplies a too-small dataset for free. Cheap partial version: pick the target pitch at random, independent of the clip, with a formant-preserving shift.
+- **Voice transfer is the real goal:** a reference of someone speaking, and the generated voice adopts it. Voice identity is time-invariant, so it fits. Harder than pitch, for the same reason the 0.75 hole exists: the generated face carries a voice prior the audio reference has to override. The clean test is two variants on the same voice data, audio-only reference versus image+audio. Pitch is the leading indicator: if pure-audio pitch doesn't move, pure-audio voice almost certainly won't.
+- **Don't chase a time-varying contour.** A negative-RoPE reference sits in a disjoint region with no alignment to the target timeline, so there is nothing to learn a moment-to-moment mapping from. Time-varying control (prosody, rhythm) wants an *aligned* driving channel at positive positions, the way LipDub drives lips frame by frame. Different experiment.
+- **If the data is fine but the adapter is just too weak,** the lever is the model side: add the cross-modal bridges (`audio_to_video_attn`, `video_to_audio_attn`), not more data.

ltx2_audio_reference.yaml

@@ -6,9 +6,9 @@
 # attribute of the generated audio+video. The reference is the CONTROLLER
 # (transfer paradigm): content varies, the attribute is read from the reference,
 # the caption is neutral. First proof = a voiced TONE reference carrying pitch ->
-# generated speech F0 tracks the tone (base-absent, so a clean YES/NO).
+# generated speech F0 tracks the tone.
 #
-# WHY THIS SHAPE (the seesaw — read before changing the pairing/targets):
+# WHY THIS SHAPE (the seesaw - read before changing the pairing/targets):
 #   "matched content" and "audio controls the attribute" are mutually exclusive.
 #   If reference and target share content, the per-example attribute must ride the
 #   caption -> TEXT controls it. For the audio to be load-bearing, reference and
@@ -47,7 +47,7 @@ lora:
   alpha: 32
   dropout: 0.0
   # AUDIO-STACK ONLY. The reference->target mixing happens in the audio
-  # SELF-attention (audio_attn1) over the [target | reference] audio sequence —
+  # SELF-attention (audio_attn1) over the [target | reference] audio sequence -
   # that is the load-bearing module. audio_attn2 (text cross-attn) + audio_ff
   # round out the audio stack. No cross-modal bridges: pitch transfer is
   # audio->audio; we do not need audio<->video coupling for this gate.
@@ -88,7 +88,7 @@ acceleration:
   mixed_precision_mode: "bf16"
   quantization: "int8-quanto"
   load_text_encoder_in_8bit: true
-  # The run+fit smoke measured an 8.70 GB training peak at 36/48 swap — huge
+  # The run+fit smoke measured an 8.70 GB training peak at 36/48 swap - huge
   # headroom (startup ~17.8 GB is the tight point). 24/48 keeps resident well
   # under that while cutting the per-step PCIe cost vs 36 (the 14 s/step at 36
   # was swap-bound). Lower toward ~16 for more speed if VRAM allows.