haohw/cosmos3-tactile-mot-v2-step3000

Cosmos-3 Tactile-MoT v2 — step-3000 (final) checkpoint

Final weights of the corrected v2 full-SFT run (3000 steps, ~10.4 h on 8×A100-80GB). Joint video + tactile future prediction on the real 15.9 B NVIDIA Cosmos3-Nano backbone: past video + past tactile → future video + future tactile (30 Hz source / 10 Hz slow / 30 Hz fast).

Results (vs the fresh, untrained backbone; official UniPC sampler)

• Future tactile forecast (normalized normal_load, RMSE vs GT): ≈10× lower — 0.106 / 0.134 at step 3000 vs 1.131 / 1.107 fresh (two fixed validation windows).
• Future video: coherent across all future frames and all three views (head + both wrists); the close-up wrist views sharpen markedly vs the fresh model; no drift/smear, no checkerboard.

See the full report in the repo: docs/cosmos3_tactile_mot_v1/reports/v2_sft_results.md, and visual GT-vs-pred montages under docs/cosmos3_tactile_mot_v1/visualizations/coherence_check/.

The tactile RMSE is a 2-window illustrative check. A rigorous evaluation (persistence-baseline gate, tactile-alignment controls, onset/release AUPRC, multiple seeds) should be run on the held-out test split — that's what this checkpoint + the splits dataset are published for.

What was trained (v2, full SFT)

Full SFT of the Cosmos3 generation pathway (~6.98 B, FSDP2/ZeRO-3, fp32-master + bf16 compute) + a fresh tactile modality; und/reasoner tower and Wan VAE frozen. Fixes over v1: aligned-10 Hz slow tactile, per-view identity embedding, English text conditioning (official Cosmos3 prompt format), official UniPC flow sampler — on top of the v1 fixes (distinct per-view mRoPE, corrected VAE conditioning boundary, removed fast-path future leakage). Both the slow expert (gen pathway + tactile head) and the 30 Hz fast expert (codec + FastTactileExpert) are trained.

Files

• model.safetensors — full model weights in bf16 (Cosmos3 backbone + tactile head + view embedding + tactile codec + fast expert + Wan VAE), ~32 GB.
• config.json — the JointModelConfig used (set snapshot to your local Cosmos3-Nano base path).
• keys.json — exact state-dict keys (load loads with missing=0 unexpected=0).

Load & evaluate

Requires the model code + the Cosmos3-Nano base snapshot.

import torch
from safetensors.torch import load_file
from tacwam.cosmos_tactile.joint_model import JointModelConfig, TactileCosmosModel

cfg = JointModelConfig(snapshot="/path/to/Cosmos3-Nano", train_mode="gen_sft", use_lora=False,
                       master_dtype="bfloat16", slow_ctx=8, slow_hor=8, num_views=3, use_fast_expert=True)
m = TactileCosmosModel(cfg).to("cuda").eval()
missing, unexpected = m.load_state_dict(load_file("model.safetensors"), strict=False)
assert not unexpected and len(missing) < 8

Or: python scripts/cosmos3/smoke_coherence.py --ckpt <dir-with-model.safetensors> --sampler unipc.

• Code: github.com/haohww/tacWAM
• Data splits (which recordings to eval on): haohw/tacwam-tactile-mot-v2-splits
• Base model: NVIDIA Cosmos3-Nano (OpenMDW-1.1); this is a derivative and inherits that license.
• Training corpus is egocentric human-subject vendor video; only weights are shared here.