VTWM ablation — short 16-frame rollouts (auto-uploaded)

Diffusion-forcing world model rollouts. Each sample is ~2.7 s (16 raw frames @ 6 fps). Side-by-side videos show ground truth on the left, model prediction on the right. First half of each clip = history (GT), second half = autoregressively sampled future. Click a sample header to expand its videos (default-collapsed to keep the page light).

Sections

Setup comparison

Click a run name to jump to its rollout videos. ✓ = feature on, ✗ = off. Lower val_loss is better.

Rollout-latent metrics (10 val samples)

Latent-space MSE of predicted future vs ground truth. Contact/no-contact split uses tactile latent-to-reference energy (threshold 0.05).

Training loss curves & conclusions

📈 Live interactive curves: https://wandb.ai/yuxiangma/tactile-world-model-diffusion (wandb — always current; static PNG below is a snapshot)

# Training loss curves — conclusions

Regenerated from wandb (`scripts/plot_loss_curves.py`). Runs covered: **37**.

- **shift16 + delta-to-ref** are the two tactile fixes; without shift16 the tactile stream diverges.
- **cam-pose** curves look fine on val_loss but decoded LPIPS is far worse — see perceptual table.
- **stride2 (4x data)** is the biggest perceptual win; p01 trades a little val_loss for rollout consistency.
- **hybrid_flow** uses rectified flow — its val_loss is on a different scale, judge by decoded metrics.

Included runs: mm_v0, mot_vt_p01, mot_vta_p01, mv1_256_delta_shift, mv1_3v2t_delta_ref, mv1_p01, mv1_p3_delta_shift, mv1_v2_perframe, mv1_v2_stride2, mv1_v2_stride2_p01_adaptw, mv1_v2_stride2_p01_hybrid, mv1_v2_stride2_p01_hybrid_deep, mv1_v2_stride2_p01_hybrid_flow, mv1_v2_stride2_p01_hybrid_rope, mv1_v2_stride2_p01_plucker, mv1_v2_stride2_p01_tactile_only, mv1_v2_stride2_p01_v2t, mv1_v2_stride2_p01_vision_only, mv1_v2_stride2_p01rand, mv2_3v2t_campose_delta_ref, mv2_p4_delta_shift, p2_mv, p3_mv2t, p4_gate, p5_gate, vo_left, vo_middle, vo_right, vo_v0, vtwm_e2_delta_ref, vtwm_e3_delta_ref_null, vtwm_n1_shift16_delta_ref_null_prenoise, vtwm_n2_shift16_delta_ref_tactile_dropout, vtwm_p0_shift16_delta_episode_p01_ref, vtwm_s1_shift16_mmdit_mv, vtwm_s2_shift16_delta_ref, vtwm_s3_shift16_delta_ref_weight

_Not on wandb yet (curve omitted): mv1_v2_h4f12, mv1_v2_stride2_p01, p4_mv2t_extr, vtwm_e1_1v2t_mmdit_mv_

A. Best stride2 baselines (window ref)

stride2 (window ref) SOTA (val_loss)

Stride=2 re-encoded latents → 4x training data. Same recipe as mv1_3v2t. Best stride2-family val_loss (0.0130). Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 + h4/f12 window

Shorter history (4), longer future (12) — 1.5x more train windows per episode. Ties stride2 SOTA at 0.0130. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

B. episode_p01 ref (rollout-consistent)

stride2 + p01

Stride2 with tactile_ref_source=episode_p01. Best perceptual quality overall (view LPIPS 0.075) despite higher latent val_loss than window. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

p01 + random no-contact ref

p01 recipe with a randomly sampled no-contact reference frame per window instead of the fixed episode_p01 frame. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

C. Architecture — MoT / hybrid

MoT V+T

Codex-designed MoT variant — per-modality QKV+MLP+adaLN with shared spatial attention pool. Functionally ≡ mmdit_mv. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

MoT V+T+Action

MoT-VT + 4 zero-init action-expert tokens joining the spatial pool. The gate never opened without a supervision signal — action tokens contributed nothing. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

stride2 + p01 + hybrid + RoPE

Dual→single hybrid fusion (HunyuanVideo/FLUX style) with RoPE positional encoding. Warm-started from mv1_v2_stride2_p01. Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

D. Single-modality controls

vision-only control

p01 recipe with the tactile stream removed — visual-only lower bound (val_loss 0.0101). Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)

tactile-only control

p01 recipe with the visual stream removed — tactile-only lower bound (val_loss 0.0062; tactile latents are easier to predict than visual). Showing 8 of 10 samples.

sample_000  (click to expand videos)

sample_001  (click to expand videos)

sample_002  (click to expand videos)

sample_003  (click to expand videos)

sample_004  (click to expand videos)

sample_005  (click to expand videos)

sample_006  (click to expand videos)

sample_007  (click to expand videos)