--- title: H3-World emoji: ๐ŸŽฎ colorFrom: indigo colorTo: green sdk: gradio sdk_version: 6.26.0 app_file: app.py python_version: "3.12" startup_duration_timeout: 1h short_description: Drive a world model with WASD and camera keys models: - MiniMaxAI/MiniMax-H3 - DANNY621/H3-World - larryvrh/MiniMax-H3-Turbo-Lora --- # H3-World โ€” action-conditioned world model [`DANNY621/H3-World`](https://huggingface.co/DANNY621/H3-World) is a rank-32 LoRA for [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3) that turns it into a **playable** world model: you give it a first frame and a **key sequence**, and it renders what happens as those keys are held. ``` forward*20, forward-right*10, pan-right-fast*7 ``` | keys | meaning | |---|---| | `W` `A` `S` `D` | walk forward / strafe left / walk backward / strafe right | | `J` `L` | camera pans left / right | | `K` `I` | camera tilts up / down | | `F` | modifier โ€” the camera move is *sharp* rather than *slow* | ## How the actions actually reach the model The checkpoint's own run manifests are unambiguous about this, and it is the thing that makes the LoRA demo-able at all: ```json "checkpoint": {"action_dim": 0, "action_mode": "text", "action_tensors": 0, "lora_tensors": 208, "lora_pairs": 104} "config": {"num_frames": 124, "height": 480, "width": 832, "fps": 24, "steps": 50, "conditioning": "first_frame+8d_actions", "action_columns": ["W", "A", "S", "D", "I", "J", "K", "L"]} ``` `action_dim: 0` and `action_tensors: 0` โ€” there is **no action encoder and no action embedding**. The 8-dimensional key state is carried through the **text channel**: one short English sentence per *latent* video frame, appended to the scene prompt, in the register the author's own runs use. So a 124-frame request (37 latent frames) is conditioned on a prompt that looks like: ``` A third-person view of a man walking through a city intersection... the man walks forward, camera follows him the man walks forward, camera follows him ... the man walks forward and strafes right, camera follows him ... the man stands still, camera pans right sharply ``` Every sentence ends in a camera clause โ€” there is no "no camera" shape in the training data. With no camera key held the reference still states what the camera is doing: `camera follows him` while the character is moving, `camera holds steady` while he is not. `caption_for()` in `app.py` is a port of the reference's `annotate_from_keys9` (`code/abot/action_script.py`), sentence-for-sentence across all 130 reachable key combinations, so the strings the model receives here are the strings it was trained on. The resolved per-frame captions are shown under **Conditioning** after every run. ## The directed attention mask The model card is explicit that the LoRA weights alone do not reproduce the reported behavior: the training run used a **directed attention mask** that binds each per-frame caption to the latents of *its* frame. Without it every video row attends to all 37 sentences at once and the sequence collapses into an average action. MiniMax-H3 is a single packed 1-D sequence under full self-attention โ€” `[text | keyframe anchors | audio | video]`, no cross-attention โ€” so the mask is a constraint inside one attention call, not a separate cross-attention mask. `app.py` reimplements it as an **exact log-sum-exp merge** rather than a dense `[S, S]` mask (which would force SDPA off its flash kernel for the whole 21k-row sequence). The keys are split into three regions: | region | keys | kernel | |---|---|---| | **A** | text rows *before* the caption block | flash, unmasked | | **C** | the ~700 caption rows | fp32 masked matmul, chunked over queries | | **B** | everything after the text block (~99% of keys) | flash, unmasked | Each returns its output and its log-sum-exp; the three are recombined with the online-softmax identity, which is numerically identical to one masked softmax over the full row. **What "directed" means.** The reference's `mask_mod` cuts the annotations' *outgoing* edges, and only those: * a sentence row, **as a key**, is readable by itself and by the video rows of its own latent frame โ€” the static prompt, the keyframe anchors, the audio and **every other sentence** are blocked from reading it; * a sentence row, **as a query**, reads the video rows of its own latent frame only. It still reads the static prompt, the anchors and the audio freely โ€” that is information flowing *into* the sentence, giving it scene grounding, not a bypass around frame *i*. Blocking `A_j โ†’ A_k` is the load-bearing half. With it, `V_k` is a cut vertex between `A_k` and the rest of the sequence; without it `A_j`'s content reaches `A_k` first, and `V_k` reading `A_k` is reading `A_j` too. The mask is toggleable in **Advanced** so you can see the difference. With it off, the same script produces a video that drifts through a blur of every action at once. Three guards, all in `app.py`: * The token spans are located by re-tokenizing prefixes of the prompt. If a BPE merge straddles a sentence boundary the spans would be wrong, so `build_conditioning_text()` verifies `cuts[-1] == total` and **refuses to mask** rather than mask the wrong rows. * Once the mask actually bites, most rows see *no* sentence key at all, so the masked region's log-sum-exp is `-inf` for them. The merge treats that as weight zero, but `softmax` over an all-`-inf` row is `NaN` โ€” so `_masked_lse` zeroes those rows explicitly. * `MiniMaxH3TokenRefinerBlock` runs the same attention module over the *text stream alone*, which the processor detects by its length (`cap_end` is the whole text length by construction). That stream is attended **block-diagonally** โ€” one segment for the scene prompt, one per sentence โ€” matching the reference's split `refiner_cu`. Running it as a single document lets sentences mix *before* the DiT backbone, and no mask on the DiT side can close an upstream leak. The LoRA adapts `token_refiner.refiner_blocks.{0,1}`, so those deltas were trained under this segmentation. ### Known divergence from the reference One difference remains, and it is upstream of everything above. The reference encodes **each sentence independently** through the text encoder and concatenates the row blocks after the head, caching by string so a repeated sentence is bit-identical. This Space builds one prompt string and sends it to the shared `multimodalart/qwen3vl-conditioner`, so sentence *k* is contextualised against the image and every preceding sentence before the DiT ever sees it. Closing it needs a raw text-encode endpoint on the conditioner, which it does not currently expose. Related: the reference also gives annotation rows a mirrored positional offset (`origin = text_len - s_k[-1] - 1`), which is not reachable from this side of the split either. ## Architecture โ€” why two Spaces MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage, so the pipeline is split at its `text_encoder` step, exactly as in [`multimodalart/minimax-h3`](https://huggingface.co/spaces/multimodalart/minimax-h3): * the 62.14 GiB Qwen3-VL conditioner runs in [`multimodalart/qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner), which this Space calls over the gradio API for every request; * this Space holds the 61.73 GiB transformer (with H3-World merged into it) plus the video and audio autoencoders. The wire format between them is `prompt_embeds` `(1, N, 5120)` bf16 + `text_token_tags` `(N,)` int64 in a single safetensors file. The caption spans this Space needs are recovered from it by offset arithmetic, because `MiniMaxH3TextEncoderStep` appends the prompt **verbatim** โ€” no chat template, no special tokens โ€” so `offset = num_text_tokens - num_prompt_tokens`. `h3_split_blocks.py` is the blockset with the `text_encoder` step removed, copied from that Space. ## LoRA merge The LoRA is published against the **original** MiniMax-H3 layout, not the diffusers port, so `load_lora_weights()` does not apply. `load_and_apply_lora()` replays `convert_minimax_h3_to_diffusers.py`'s renames on the way in โ€” `blocks.` โ†’ `transformer_blocks.`, `attn.out_proj` โ†’ `attn.to_out.0`, `mlp.fc1` โ†’ `ff.net.0.proj` with the SwiGLU gate/value halves swapped, and the fused `attn.qkv_proj` de-interleaved per head before being split into `to_q` / `to_k` / `to_v`. 104 LoRA pairs become 208 merged weight deltas; **any target that fails to resolve is fatal**, never skipped. ## 28 steps vs the 8-step turbo LoRA **Sampling** in the UI picks between two configurations of the same request โ€” same seed, same action script, same directed mask โ€” so the quality cost of the distillation is directly visible: | mode | steps | transformer | |---|---|---| | `28 steps ยท no turbo LoRA` | 28 (MiniMax-H3's default) | H3-World only | | `8 steps ยท turbo LoRA` | 8 | H3-World **+** [`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora) `minimax_h3_turbo_v4_step600_ema.safetensors` | `h3_turbo_lora.py` folds the repo's own recommended checkpoint โ€” the `v4` line, step 600, EMA: * the file is a **ComfyUI-side checkpoint in the original MiniMax-H3 naming** (`blocks.N.attn.qkv_proj.lora_A.weight`, `blocks.N.adaln_proj.linear`, `final_layer.adaln_proj.linear`), so like H3-World it is replayed through `convert_minimax_h3_to_diffusers.py`'s renames: **259 LoRA pairs become 363 weight deltas**, covering `to_q`/`to_k`/`to_v`/`to_out.0`/`ff.net.0.proj`/`ff.net.2` on all 52 attention blocks plus the 50 block-level AdaLN projections and `norm_out.linear`; * one transform differs from the H3-World merge: the adapter was trained against `comfy.ldm.minimax.model`, whose attention reads `qkv_proj(x).split(heads * head_dim, dim=-1)`, so `Comfy-Org/MiniMax-H3` holds the fused QKV as contiguous `[q_all; k_all; v_all]` and these rows are split into thirds **without** the raw-shard per-head de-interleave. The `mlp.fc1` `[gate; value]` โ†’ `[value; gate]` swap still applies, and both AdaLN projections are pure renames; * ranks are mixed by design โ€” 64 on the attention/FFN projections, 16 on the AdaLN ones โ€” and the file's metadata says `application: W_eff = W + lora_B @ lora_A` with the card's *"alpha = rank, so no extra scaling"*, so the fold scale is exactly **1.0** and `H3_TURBO_STRENGTH` is left as the card's strength dial; * the card's useful range is **4โ€“8 steps** (6โ€“8 recommended, nothing gained past 8), so the step count is overridden to **8 NFE**. No scheduler swap and no CFG change: MiniMax-H3 is already guidance-distilled and this Space keeps its native `MiniMaxH3Scheduler` with the turbo LoRA folded. It is *folded* into the bf16 weights, like H3-World, because this Space patches `MiniMaxH3AttnProcessor` and drives the transformer's live weights. Fold and unfold are the same operation with a sign, so the low-rank factors stay resident and `set_active` flips the mode in place inside the `@spaces.GPU` call, through one bf16 rounding. The Steps slider still overrides the mode's count (4โ€“50), so `50 steps + turbo LoRA` or `8 steps without it` are both reachable for the sake of the comparison. ## Generation constraints Fixed by the checkpoint: 24 fps, `num_frames` snapped to `17n + 5`, no CFG and no negative prompt (it is guidance-distilled). H3-World was trained at **832x480**; the conditioner's canvas list does not offer that exact size, so the default here is its nearest neighbour, **960x544**. The offered canvases are the cheap tier of each aspect ratio rather than the conditioner's full list. The mask term scales as *sequence x caption rows*, so a 1344x768 / 8 s request would want ~35 GPU-minutes โ€” past what any visitor could book โ€” and it is off-distribution for a LoRA trained at 832x480 anyway. ## Measured On this Space, driven over `gradio_client`, at 960x544 with a keyframe: | Request | Conditioner | Denoise + decode | Round trip | |---|---|---|---| | 16 steps, 56 frames, directed | 2 s | 45 s | 50 s | | 16 steps, 56 frames, **no mask** | 2 s | 32 s | 36 s | | 50 steps, 124 frames, directed (the default) | 10 s | 303 s | 316 s | Startup is 95 s: the 66.3 GB download, the load, the LoRA merge, and the ZeroGPU pack. `get_duration` is fitted to exactly these three points โ€” the unmasked block cost linear + quadratic in the packed sequence, the mask's own term linear in *sequence x captions* โ€” and books ~15% over the fit. The default request books 348 s. ## Examples The three bundled first frames are extracted from [`acvlab/ABot-World-Explorer-500h`](https://huggingface.co/datasets/acvlab/ABot-World-Explorer-500h) (Apache-2.0), which is the same kind of third-person game footage H3-World was trained on. Each is paired with that clip's own manifest prompt. ## Space variables | Variable | Default | Meaning | |---|---|---| | `H3_MODEL_REPO` | `MiniMaxAI/MiniMax-H3` | The diffusers-layout base checkpoint. | | `H3_LORA_REPO` | `DANNY621/H3-World` | The LoRA. | | `H3_LORA_FILE` | `step-10000.safetensors` | The checkpoint the author's own test runs used. | | `H3_TURBO_REPO` | `larryvrh/MiniMax-H3-Turbo-Lora` | The turbo-LoRA repo. | | `H3_TURBO_FILE` | `minimax_h3_turbo_v4_step600_ema.safetensors` | The repo's recommended `v4` checkpoint. | | `H3_TURBO_STEPS` | `8` | Steps the turbo mode asks for (the card's range is 4โ€“8). | | `H3_TURBO_STRENGTH` | `1.0` | The card's strength dial; `alpha == rank`, so `1.0` applies the update as-is. | | `H3_CONDITIONER` | `multimodalart/qwen3vl-conditioner` | The Space this one asks for embeddings. | | `H3_ATTENTION` | `_native_cudnn` | cuDNN's fused kernel. flash-attention 3 is sm90-only; this pool is sm120. | | `H3_GPU_SIZE` | `xlarge` | ZeroGPU allocation size. `large` does not fit. | ## License The LoRA is Apache-2.0, but usage is governed by the **base model's** license ([`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)).