mikelmh025/primitive-vace-49f-ckpts

primitive-vace-49f: checkpoints, results, data (gated)

Finetunes of Wan-AI/Wan2.1-VACE-1.3B-Diffusers (full finetune, 2.154 B params) for primitive-conditioned video generation. The model generates a video from frame 0, per-frame 3D-box depth, per-frame instance maps, and text. Code: github.com/liuwei283/agent_wm, branch nobg-nocam.

Internal research use only. data/ and results/ contain imagery derived from the Waymo Open Dataset and fall under its terms: do not redistribute.

checkpoints/          one folder per training run, step_XXXXXX/training_state.pt
results/              progress_report_2026-09-29/ (+ one-file .tar.gz of the same report)
data/                 waymo49f_vlm_captions/  (per-clip captions for all 8,387 Waymo clips)

checkpoints/

Every checkpoint is a full training_state.pt (12.95 GB): transformer weights, optimizer state and global_step. Load it with `infer_primitive_vace.py --checkpoint

` or resume with `train_primitive_vace.py --resume `; this is the non-LoRA format, `is_lora=False`. Inference settings for all of them: guidance 1.0, 50 steps, default flow shift. Guidance > 1 is harmful; see the report.

folder	background plate	dataset	recipe	steps	held-out LPIPS ↓ (n)
wan21vace_49f_fullft/	given	tabletop 49f (23k)	fixed prompt, lr 2e-5 cosine, eff-batch 32	1k–17k	0.0598 @10k (50)
wan21vace_tt49f_metaprompts/	given	tabletop 49f (23k)	per-sample detailed prompts	2k–26k	0.0594 @20k (50)
wan21vace_waymo49f_fullft/	given	Waymo 49f (7.9k)	fixed driving prompt	1k–10k	0.0842 @6k (50)
wan21vace_tt49f_nobg/	none, camera off	tabletop 49f (1.8k)	template prompt; config configs/butters_tt49f_nobg.yaml	4k	0.1709 @4k (16)
wan21vace_waymo49f_nobg_v2/	none, camera off	Waymo 49f (8k)	VLM caption, prompt dropout 0.1; configs/butters_waymo49f_nobg_v2.yaml	7k	0.1460 @7k (16), the best no-bg Waymo model
wan21vace_waymo49f_nobg_v3/	none, camera off	Waymo 49f (8k)	resumes v2@26k, object-weighted loss ×5, lr 8e-6; configs/butters_waymo49f_nobg_v3_continue.yaml	33k, 45k	0.1472 @33k, 0.1515 @45k (16)

Background-given and no-background numbers are not comparable. The no-bg model has to synthesise the whole background from frame 0, while its object regions match the bg-given model (object PSNR 22.95 vs 22.81 dB). v3@45k is the final checkpoint and the one the report examples use. It is worse than v3@33k on 15/16 clips and tied with v2@7k. Each new run folder includes its training config (train_config.yaml). Every other no-bg checkpoint (every 500–1000 steps, plus EMA weights for v3) is on the training box and can be uploaded on request.

results/progress_report_2026-09-29/

The team progress report. Start with its README.md. It contains:

• 8 Waymo and 8 tabletop examples, each with GT, generated video and every conditioning input.
• The Waymo labeling-pipeline write-up with worked examples.
• The evaluation summary of all experiments, with the original metric files.

project_progress_report_2026-09-29.tar.gz is the same report as one download.

data/waymo49f_vlm_captions/

captions_bundle.json holds all 8,387 captions. packets/shard_XXXX.tar holds the per-clip labeler packets, one tar per Waymo shard (17 tars, ~590 MB each). They are packed because HF limits a repo to 20k files. Each tar unpacks to <shard>_<sample>/ with the labeler inputs (facts.json, sheet.png, crops.png) and output (caption.json, with the labeler's notes). pipeline/ has the spec and tooling. Each caption comes from a single labeler with no cross-model verification; see its README.md for the quality caveats.