Upload README.md

README.md

@@ -1,5 +1,5 @@
 ---
-title: MiniMax-H3 · reference · Custom lora + CivitAI links, GPU cost, profiles, clip stitching
+title: MiniMax-H3 · reference → video + audio · Custom lora
 emoji: 🎭
 colorFrom: pink
 colorTo: purple
@@ -7,7 +7,7 @@ sdk: gradio
 sdk_version: 6.20.0
 app_file: app.py
 pinned: true
-short_description: Video + soundtrack, your lora + CivitAI, cost, profiles
+short_description: Video + soundtrack, your lora + CivitAI, structured prompts
 suggested_hardware: zero-a10g
 tags:
   - video
@@ -26,27 +26,49 @@ audio references, at **bfloat16 with no quantization anywhere**.
 
 This Space is the denoising half of the `ref2va` task: the 61.73 GiB `transformer_ref` partition and the two
 autoencoders. The 62.14 GiB Qwen3-VL conditioner runs in
-[`qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner), which this
-Space calls over the gradio API for every request — the same conditioner Space, and the same resident weights, that
-the keyframe half [`minimax-h3`](https://huggingface.co/spaces/multimodalart/minimax-h3)
-uses.
+[`qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner), which this Space calls over
+the gradio API for every request — the same conditioner Space, and the same resident weights, that the keyframe half
+[`minimax-h3`](https://huggingface.co/spaces/multimodalart/minimax-h3) uses.
 
 ## What this fork adds
 
-**Custom lora, three slots.** A Hugging Face repo, a file inside one, a **CivitAI download link**, or an uploaded
+**A structured prompt builder.** H3 was trained on the output of H3-Context-IR, a preprocessor that rewrites a plain
+request into labelled sections, and MiniMax's own model card calls that structure *critical to the quality of the
+final output*. Nothing in this pipeline adds it — the string reaches the transformer as typed. The builder writes
+those sections for you: the reference line naming `<Picture 1>`, `<Picture 2>` in connection order, then
+`integrated_multimodal_description` with the shot type, the camera move written as type + amplitude + speed, and the
+dialogue verbatim inside `<d>` tags, then `overall_soundscape` and `non_diegetic_music`. Speech is generated together
+with the picture, so naming that someone speaks without giving the words produces correct mouth shapes with nothing
+in them; the builder makes that hard to get wrong.
+
+**Custom lora, five slots.** A Hugging Face repo, a file inside one, a **CivitAI download link**, or an uploaded
 file. Adapters have to be trained against the `transformer_ref/` partition — a `transformer/` adapter is a different
 partition and will not match.
 
-**kohya / CivitAI files are converted on the fly.** Most of what CivitAI carries is kohya-named, and it differs from
-diffusers in ways that quietly ruin a result rather than raise: flat underscored names, a fused QKV interleaved *per
-attention head* (a plain three-way split hands q's rows to k), a gated MLP whose two halves sit in the opposite
-order, and an `alpha` that sets the scale. All four are handled at load time, and `.pt` files are read as well as
-`.safetensors`.
-
-**Turbo presets.** Few-step distillations that render joint video + soundtrack in 4–8 steps instead of the usual
-~28. Picking one fills a free slot at strength `1.0` and moves the steps slider to the count it was distilled for.
-
-**Clip stitching, soundtrack included.** Queue several clips and they are joined into a single file; tick the box and
+**CivitAI, searched from inside the Space.** Type a word, get the model's title, version, file, size, downloads and
+trigger words, and drop the one you want straight into a slot. A link that is already in a slot can be named with
+one press, so a row reads *Hairy dad butt · v1.0 · hairy.safetensors* rather than a number. Gated entries need
+`CIVITAI_TOKEN` under *Settings → Variables and secrets*: the Space downloads on its own machine, so being signed in
+to CivitAI in a browser does not authenticate it — a gated model answers a server with an HTML login page, which is
+caught and reported rather than saved as weights.
+
+**kohya, CivitAI and LoKr files are converted on the fly.** Most of what CivitAI carries is kohya-named, and it
+differs from diffusers in ways that quietly ruin a result rather than raise: flat underscored names, a fused QKV
+interleaved *per attention head* (a plain three-way split hands q's rows to k), a gated MLP whose two halves sit in
+the opposite order, and an `alpha` that sets the scale. LyCORIS **LoKr** goes further and stores each layer as the
+Kronecker product of two small factors, which PEFT cannot load at all; it is rebuilt into an ordinary low-rank pair
+at load time — exactly, since the SVD of a Kronecker product is the outer product of the factors' SVDs, so nothing
+the size of the full 7168×7168 layer is ever built. `.pt` files are read as well as `.safetensors`.
+
+**Turbo presets.** Few-step distillations that render joint video + soundtrack in 4–8 steps instead of the usual ~28.
+Picking one fills a free slot at that build's own strength and moves the steps slider to the count it was distilled
+for.
+
+**Continue the scene.** One press takes the last frame of the clip you have, makes it the first reference, generates
+again with the settings untouched, and joins the two into a single file — soundtrack included. Press again for a
+third clip. Each press costs one normal generation; the joining is free.
+
+**Clip stitching, soundtrack included.** Queue several clips and they are joined into one file; tick the box and
 every new generation joins on its own. It runs on the CPU, so it costs nothing from the GPU allowance. Everything is
 scaled to the first clip's frame, and a clip without audio gets silence rather than breaking the join.
 
@@ -58,10 +80,6 @@ so a request that will not fit is visible as a number rather than as a failed ru
 **Randomize seed**, drawn fresh per press and written back into the box, so the number shown is always the one the
 clip was made with.
 
-Set `CIVITAI_TOKEN` under *Settings → Variables and secrets* for gated CivitAI models: the Space downloads on its own
-machine, so being signed in to CivitAI in a browser does not authenticate it — a gated model answers a server with an
-HTML login page, which is caught and reported rather than saved as weights.
-
 ## Why split
 
 MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at **150 GB of storage**. An unquantized single
@@ -80,9 +98,7 @@ reads them**. The order is semantic: it numbers the labels of MiniMax-H3's promp
 `<Video 1>`, `<Audio 1>`) and it advances the shared audio/video rotary clock, so the same references in a different
 order are a different request. This demo lays the slots out as one tab per modality in reading order — images, then
 audio, then video — and assembles the request that way. The Images tab opens with two slots and **+ Add another
-image** reveals the rest, up to the model's own nine; the audio and video tabs hold one each. A reference left in a
-tab that is not the open one is still part of the request; the tabs lay the slots out, they do not choose between
-them.
+image** reveals the rest, up to the model's own nine.
 
 Rules the model imposes, enforced here before anything is uploaded:
 
@@ -92,108 +108,23 @@ Rules the model imposes, enforced here before anything is uploaded:
   which is why the duration slider disappears when a single reference can set it, and comes back when two can or
   when the one that could is out of range.
 
-## Example assets
-
-`examples/subject.png` is [a studio portrait by Oliver
-Dohrn](https://www.pexels.com/photo/casual-portrait-of-smiling-woman-in-studio-31747504/) cropped to head and
-shoulders — Pexels licence, free to use. `examples/motion.mp4` is a synthetic clip from the parity fixtures, and
-`examples/voice.wav` is utterance `1462-170145-0022` of
-[LibriSpeech](https://www.openslr.org/12) `dev-clean` — CC BY 4.0, read from a public-domain LibriVox recording. It
-is 16 kHz mono on purpose: the audio VAE wants 32 kHz, so the example exercises the `torchaudio` resample the
-`ref2va` path needs.
-
-## How the split is expressed
-
-`MiniMaxH3Blocks` is one `SequentialPipelineBlocks` whose branches are picked per request — and per `workflow=` —
-from the inputs, `ref2va` being the branch `references` selects:
-
-```
-setup -> text_encoder -> reference_encoder -> denoise -> after_denoise -> decode
-```
-
-where `denoise` is itself `prepare_layout -> prepare_latents -> set_timesteps -> denoise`, against the
-`transformer_ref` partition.
-
-`h3_split_blocks.py` subclasses it with the `text_encoder` step removed. Dropping the step drops the three
-components it declares, so `load_components` resolves `transformer_ref` / `vae` / `audio_vae` / the two schedulers
-out of the shared `modular_model_index.json` and never fetches the conditioner — and `prompt_embeds` and
-`text_token_tags` become ordinary required inputs of the pipeline call:
-
-```py
-pipe = MiniMaxH3Ref2VAGeneratorBlocks().init_pipeline("MiniMaxAI/MiniMax-H3")
-pipe.load_components(dtype=torch.bfloat16)
-state = pipe(prompt_embeds=..., text_token_tags=..., references=[...], height=544, width=960, num_frames=124,
-             num_inference_steps=28)
-```
-
-Only **text** encoding is remote. `reference_encoder` is the `ref2va` branch's own encoder step — it runs the video
-VAE over the image and video references and the audio VAE over the soundtracks, and it is where the references'
-latent geometry is resolved — so it stays on this side, next to the autoencoders the conditioner Space does not hold.
-
-The wire format is the same two tensors as the keyframe half: `(1, num_text_tokens, 5120)` bfloat16 and
-`(num_text_tokens,)` int64, carried as one safetensors file with the resolved `height` / `width` / `num_frames` in
-its metadata header. What differs is only what the conditioner is shown, so the references travel to it as files:
-`ref2va`'s presentation puts a vision block in front of the prompt for every image and every merged video frame
-pair. An audio reference contributes its `"<Audio j>: "` label and nothing else — a waveform never reaches the
-conditioner — but it still goes over, because a single audio-bearing reference is what resolves `num_frames` when
-the request leaves it open.
-
-The `setup` step runs on **both** halves. It owns no component (PIL, PyAV-decoded media and arithmetic) and it
-resolves the canvas, the `17 * n + 5` frame count and the references prepared at their own resolutions. It is
-deterministic over the same files, and the conditioner returns the plan it resolved so this Space pins the same
-canvas and frame count rather than re-deriving them.
-
-## AoTI-compiled blocks
-
-With `H3_AOTI=1` the 50 repeated transformer blocks run from a compiled package,
-[`multimodalart/minimax-h3-aoti`](https://huggingface.co/multimodalart/minimax-h3-aoti)`:bf16/torch2.11/sm120/dynamic` — a single dynamic-sequence artifact that serves
-every canvas, duration, reference set and prompt length.
-
-It is the **same package the `transformer/` partition runs**, and nothing about it is partition-specific. The two
-`config.json` files are identical field for field, and the package carries no weights at all: `LazyAOTIModel` binds
-each block's own live `state_dict()` by name on its first forward. Patching it in is startup CPU work and costs no
-GPU time.
-
-It removes a near-constant ~0.5 s/step — 50 blocks' worth of kernel-launch overhead plus the norm / rotary / AdaLN
-epilogues around the matmuls — and cannot touch the matmuls themselves, so it pays best where the block is not
-compute bound. `ref2va` packs the reference rows in front of the generated ones, which makes the sequence longer
-than a keyframe request at the same canvas and moves it further toward compute bound.
-
-## Nothing is paid for with GPU time
-
-The 77.3 GB download and the load happen at **startup**: `import spaces` at module top patches `torch.cuda` before
-any GPU is attached, so nothing about the load needs a card. The conditioner round trip is a network call on this
-Space's CPU. A `@spaces.GPU` call is therefore only the placement (once), the two reference encoders, the denoise
-loop and the two decoders.
-
-One thing does *not* happen at startup: the move onto the card. `spaces`' startup `torch.pack()` writes every
-startup-resident CUDA tensor to a second copy on disk before deleting the downloaded originals, and 77.3 GB of
-weights plus a 77.3 GB pack is 154.6 GB against a 150 GB quota — the Space is evicted mid-pack with `OSError:
-[Errno 28] No space left on device`. Placement therefore happens on the **first GPU call**, `PIPE.to("cuda")` at the
-top of the `@spaces.GPU` function: about 10 s of PCIe once, then a no-op walk, and the denoise loop runs with
-everything resident and no offloading at all.
-
-The references are decoded inside that call too, from their paths rather than as decoded media. A `@spaces.GPU`
-argument crosses a process boundary by pickling, and a 5 s 1344x768 reference video is 370 MB of frames once PyAV
-has expanded it.
-
 ## Generation constraints
 
 Fixed by the checkpoint: 24 fps, a 768 pixel short edge, 5 to 15 s, `num_frames` snapped up to the next `17 * n + 5`,
 no CFG and no negative prompt (it is guidance-distilled, so every step is one forward pass). The duration slider
-stops at 14 s because it is the *snapped* count that has to hold for the ceiling: 15 s is 360 frames, which rounds
-up to 362, i.e. 15.083 s, and is refused.
+stops at 14 s because it is the *snapped* count that has to hold for the ceiling: 15 s is 360 frames, which rounds up
+to 362, i.e. 15.083 s, and is refused.
 
 ## GPU time is reserved per request, not per Space
 
-MiniMax-H3 attends over one packed sequence, so what a step costs is a function of that sequence's length alone —
-and on this half the references dominate it. A single 1344x768 image reference is ~7168 conditioning rows plus the
-vision block it puts in front of the prompt; a 2.5 s video reference is another ~17000. The same 960x544, 124-frame
-request runs 2.4 s/step with no references and 16 s/step with an image and a video.
+MiniMax-H3 attends over one packed sequence, so what a step costs is a function of that sequence's length alone — and
+on this half the references dominate it. A single 1344x768 image reference is ~7168 conditioning rows plus the vision
+block it puts in front of the prompt; a 2.5 s video reference is another ~17000. The same 960x544, 124-frame request
+runs 2.4 s/step with no references and 16 s/step with an image and a video.
 
-`get_duration` prices that before the call instead of reserving a flat ceiling for everything. It takes the
-arguments of the `@spaces.GPU` function, so it has the conditioner's own `text_token_tags` (exact) and the reference
-files (measured from metadata, no decode), and evaluates
+`get_duration` prices that before the call instead of reserving a flat ceiling for everything. It takes the arguments
+of the `@spaces.GPU` function, so it has the conditioner's own `text_token_tags` (exact) and the reference files
+(measured from metadata, no decode), and evaluates
 
 ```
 S = text rows + reference rows + target rows
@@ -204,8 +135,12 @@ fitted on the `t2va` half and checked against live `ref2va` requests to about 10
 pool reserves whatever number it is given, and a flat 900 s is what makes a busy account fail admission with *"You
 have too many ZeroGPU credits allocated to running tasks."* A typical single-image request now reserves ~460 s.
 
-Every request carries the full placement allowance, because nothing on this side knows whether the worker it lands
-on is cold and a cold one pays the lazy 72.16 GiB `PIPE.to("cuda")` inside its first GPU call.
+## Nothing is paid for with GPU time that does not have to be
+
+The 77.3 GB download and the load happen at **startup**: `import spaces` at module top patches `torch.cuda` before
+any GPU is attached, so nothing about the load needs a card. The conditioner round trip is a network call on this
+Space's CPU. So is every lora download, every CivitAI lookup, and all clip stitching. A `@spaces.GPU` call is only
+the placement (once), the two reference encoders, the denoise loop and the two decoders.
 
 ## Space variables
 
@@ -215,47 +150,22 @@ on is cold and a cold one pays the lazy 72.16 GiB `PIPE.to("cuda")` inside its f
 | `H3_MODEL_REPO` | `MiniMaxAI/MiniMax-H3` | The diffusers-layout checkpoint. Public. |
 | `H3_AOTI` | `0` | `1` loads the compiled block package. |
 | `H3_PLACEMENT` | `lazy` | `lazy` moves all 72.16 GiB onto the card on the first GPU call and leaves it there; `offload` hands placement to `ComponentsManager.enable_auto_cpu_offload` instead. |
-| `H3_ATTENTION` | `_native_cudnn` | cuDNN's fused kernel, 10–20% faster than the SDPA default and needs nothing installed. flash-attention 3 is sm90-only and this pool is sm120. The two VAEs are pinned to torch SDPA instead: they are float32, which cuDNN has no kernel for. |
+| `H3_ATTENTION` | `_native_cudnn` | cuDNN's fused kernel, 10–20% faster than the SDPA default. flash-attention 3 is sm90-only and this pool is sm120. |
 | `H3_GPU_DURATION_MIN` / `_MAX` | `120` / `1500` | Bounds on what `get_duration` may reserve. |
 | `H3_PLACEMENT_ALLOWANCE` | `90` | Seconds of the reservation set aside for a cold worker's placement. |
 | `H3_GPU_SIZE` | `xlarge` | ZeroGPU allocation size. `large` does not fit. |
-| `CIVITAI_TOKEN` | unset | Appended to CivitAI download links, for models that are not open. |
-
-## Whose GPU quota pays
-
-Two cards are booked per request — this Space's denoise loop and the conditioner's forward — and both are billed to the
-requesting user, with nothing here arranging it: `gradio_client` attaches the caller's own `x-ip-token` to every
-outgoing call, reading it off gradio's `LocalContext` inside the event listener (`Client.send_data` ->
-`add_zero_gpu_headers`), and ZeroGPU charges the booking to whatever that token identifies.
-
-A caller with no token to forward — a `gradio_client` script rather than a browser — leaves the conditioner's booking
-attributed to this Space's pod IP and its small shared quota. An unattributed caller may book at most 120 credits at a
-time and an `xlarge` booking costs twice its seconds, so the conditioner books the encode (45 s) and a prompt upsample
-(60 s) as two separate calls, each within that ceiling.
-
-## Secrets
-
-None are required for generation. Everything this Space downloads is public — the
-[`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3) checkpoint and the
-[`multimodalart/minimax-h3-aoti`](https://huggingface.co/multimodalart/minimax-h3-aoti) packages — and the conditioner
-is a public Space called on the requesting user's own ZeroGPU token, never on an org token. `CIVITAI_TOKEN` is the one
-optional addition, and only for a gated adapter.
+| `H3_LOKR_RANK` | `32` | Rank the LoKr converter targets. Raise it if the log reports a weak layer. |
+| `CIVITAI_TOKEN` | unset | Used for CivitAI downloads and search, and required for gated or adult entries. |
+| `CIVITAI_API_HOST` | unset | Pins the search host; otherwise `civitai.red` is asked first and `civitai.com` is the fallback. |
 
 ## Where diffusers comes from
 
 MiniMax-H3 is modular-only and not in a released `diffusers`, so `requirements.txt` installs it from the canonical
 pull request, [huggingface/diffusers#14371](https://github.com/huggingface/diffusers/pull/14371), pinned to the commit
-`665f5782` (`refs/pull/14371/head`) rather than to the moving `minimax-h3-refactor` branch.
-
-That PR is a WIP, so it needs re-pinning whenever it updates, and `h3_split_blocks.py` — which subclasses its block
-classes to cut the pipeline in two — has to be re-checked against the new head at the same time.
-
-Two of those are `ref2va`-only and easy to miss. PyAV decodes a reference video or audio file as the reference is
-built, and **`torchaudio`** resamples a soundtrack that is not already at the audio VAE's 32 kHz — a 32 kHz
-reference skips the resample entirely, so the dependency only shows up once someone brings audio at another rate:
+`665f5782` rather than to the moving `minimax-h3-refactor` branch. That PR is a WIP, so it needs re-pinning whenever
+it updates, and `h3_split_blocks.py` has to be re-checked against the new head at the same time.
 
-```
-ImportError: Resampling a MiniMax-H3 reference soundtrack from 24000 Hz to 32000 Hz needs `torchaudio`.
-```
+## Attribution
 
-The conditioner Space needs it as well: its `setup` step normalizes the very same waveforms this one does.
+An optimized derivative of [`multimodalart/minimax-h3`](https://huggingface.co/spaces/multimodalart/minimax-h3).
+MiniMax-H3 weights remain governed by the MiniMax-H3 Community License Agreement.