Run full causal Geometry Forcing RE10K evaluation at 12 slots and CFG1

Extend the tested causal evaluator to eight workers and 300-frame loops with separate worker records. The full launcher evaluates one monitor preflight, then all100fixedscenes with common scoring and aggregation. Teacher-history diagnostics are optional. Sampling code, released weights, environment and required cache paths are unchanged.

README.md

@@ -18,7 +18,7 @@ Four independent model workspaces and conda environments for the existing Minecr
 | DecMem public Wan reference | `DecMem/` | `/proj/cvl/users/x_fahkh2/envs/decmem-eval` | Minecraft: 600 observed + 500 generated frames |
 | Matrix-Game 2.0 | `Matrix-Game/Matrix-Game-2/` | `/proj/cvl/users/x_fahkh2/envs/matrix-game2-eval` | Minecraft: 600 observed + 500 generated frames |
 | LIVE | `LIVE/` | `/proj/cvl/users/x_fahkh2/envs/live-eval` | RE10K: 100 observed + 300 generated frames |
-| Geometry Forcing | `GeometryForcing/` | `/proj/cvl/users/x_fahkh2/envs/geometry-forcing-eval` | RE10K: causal diagnostics; native interpolation is an offline control |
+| Geometry Forcing | `GeometryForcing/` | `/proj/cvl/users/x_fahkh2/envs/geometry-forcing-eval` | RE10K: causal 12-slot/CFG1 evaluation; native interpolation is an offline control |
 
 The Berzelius checkout location is `/proj/cvl/users/x_fahkh2/worldmem-baseline-evals`. Submit jobs from Berzelius Hopper. First, `deployment/setup_environment_berzelius.sh` creates all four environments under `/proj/cvl/users/x_fahkh2/envs/`, builds required CUDA extensions, downloads checkpoints, validates data, and caches metric weights. It installs the official pinned Miniforge 24.7.1-2 manager under `/proj/cvl/users/x_fahkh2/envs/baseline-miniforge3` and restricts conda environment creation to the `conda-forge` channel. It uses that manager by its absolute path, without a Miniforge module or shell-profile changes. It uses the dedicated `berzelius-hopper-cpu` partition, 14 CPU cores, no GPUs, and an eight-hour limit. It runs no model inference. Setup logs are written under `slurm_logs/`, and records under `artifacts/environment_setup_JOBID/`. All setup, smoke and full evaluation jobs set both `TMPDIR` and `TRITON_CACHE_DIR` to `/proj/cvl/users/x_fahkh2/worldmem-baseline-evals/cache/`, as required by the project.
 
@@ -29,11 +29,11 @@ After setup completes successfully, each method has its own smoke job and full e
 | DecMem | `deployment/smoke_decmem_1h200.sh` | `deployment/evaluate_decmem_8h200.sh` |
 | Matrix-Game 2.0 | `deployment/smoke_matrix_game2_1h200.sh` | `deployment/evaluate_matrix_game2_8h200.sh` |
 | LIVE | `deployment/smoke_live_1h200.sh` | `deployment/evaluate_live_8h200.sh` |
-| Geometry Forcing (offline control) | `deployment/smoke_geometry_forcing_1h200.sh` | `deployment/evaluate_geometry_forcing_8h200.sh` |
+| Geometry Forcing (causal, 12 slots, CFG1) | `deployment/geometry_causal_controls_1h200.sh` | `deployment/evaluate_geometry_forcing_8h200.sh` |
 
-For the strictly autoregressive comparison, use `deployment/geometry_causal_controls_1h200.sh`. Its four separate tasks test the previous 8-slot/guidance-4 setting, 12 slots/guidance 4, 12 slots/guidance 1, and a diagnostic that replaces generated history with ground-truth history while retaining the same noise masks. Each uses two monitor scenes and 32 future frames. The three autoregressive tasks use only past RGB and poses through the current target; unused model slots contain noise and copies of the current camera. The ground-truth-history task is not eligible as a benchmark result. These short tests must be reviewed before preparing a full causal evaluation.
+The completed strictly autoregressive diagnostics used `deployment/geometry_causal_controls_1h200.sh`: 8 slots/guidance 4, 12 slots/guidance 4, and 12 slots/guidance 1 on two monitor scenes with 32 future frames. Its optional fourth task replaces generated history with ground-truth history while retaining the same noise masks; this task is diagnostic only. The autoregressive tasks use past RGB and poses through the current target; unused model slots contain noise and copies of the current camera. The full launcher uses the selected 12-slot/guidance-1 setting without waiting for the fourth task.
 
-Run smoke jobs separately and inspect prediction quality before the corresponding full evaluation. Each smoke uses one real-data case, runs generation and common scoring, then checks aggregation. LIVE generates 104 frames; Geometry Forcing generates the complete 300-frame loop to exercise both keyframe batches and interpolation. Minecraft tests generate 20 DecMem or 24 Matrix frames. Smoke logs are under the checkout's `slurm_logs/`; full logs are under each method workspace's `slurm_logs/`. Results are under each method workspace's `outputs/`.
+The full causal GF job includes its own 300-frame preflight. Standalone LIVE and Minecraft smoke jobs each generate and score one real-data case and check aggregation: LIVE generates 104 frames, DecMem 20, and Matrix 24. The separate native GF offline smoke generates 300 frames using keyframes and interpolation. Smoke and causal-diagnostic logs are under the checkout's `slurm_logs/`; full logs are under each method workspace's `slurm_logs/`. Results are under each method workspace's `outputs/`.
 
 The full launchers partition cases without duplication, score each case, and aggregate all requested outputs. Minecraft evaluates 300 cases with seeds 42 and 43 in sequential array elements; RE10K evaluates the fixed final100 scope with seed 0. Each GPU job requests 14 CPU cores per H200 and leaves memory allocation to Slurm. Each model has a separate submitted job. No script submits other jobs automatically.
 
@@ -41,8 +41,10 @@ Expected existing dataset locations are `/proj/cvl/users/x_fahkh2/WorldMem_Repro
 
 **Protocol details.** Minecraft observes source frames `[100,700)` and scores `[700,1200)` at 360×640, against the fixed Oasis-VAE reference cache. DecMem uses actions and poses with UniPC20/CFG5; Matrix uses calibrated action conditioning and its native three-step sampler. Temporal padding duplicates only observed frames, and the scored suffix contains exactly 500 frames. Matrix's incoming-to-outgoing action conversion is documented in its method manifest.
 
-RE10K follows `G0…G199,G199…G0`, including the repeated turnaround, with 100 observed and 300 generated frames at 256×256. LIVE uses its 18-step sampler with the requested 12-frame window. Geometry Forcing consumes only the last observed image (timeline 99), predicts 19 sparse keyframes over timeline 99–399, and interpolates the other frames with its native 16-slot backbone. Prediction guidance is 4 with stabilization 0.02; interpolation guidance is 1.5. Short camera batches repeat the final camera to fill unused model slots. This offline GF workflow is not a 12-slot comparison or a strictly autoregressive evaluation. Scene seeds follow `1_000_003 * metadata_index + 7`. The common RGB targets, AlexNet LPIPS, mean frame PSNR, SSIM and pooled FID are retained; these are not the paper's exact metric implementation. New GF outputs use the `geometry-forcing_native_re10k_` prefix.
+RE10K follows `G0…G199,G199…G0`, including the repeated turnaround, with 100 observed and 300 generated frames at 256×256. LIVE uses its 18-step sampler with the requested 12-frame window. The Geometry Forcing offline control consumes the last observed image (timeline 99), predicts 19 sparse keyframes over timeline 99–399, and interpolates the other frames with its native 16-slot backbone. Its prediction guidance is 4 with stabilization 0.02; interpolation guidance is 1.5. Short camera batches repeat the final camera to fill unused model slots. This offline workflow is not the causal evaluation described below. Scene seeds follow `1_000_003 * metadata_index + 7`. The common RGB targets, AlexNet LPIPS, mean frame PSNR, SSIM and pooled FID are retained; these are not the paper's exact metric implementation. Archived offline GF outputs use the `geometry-forcing_native_re10k_` prefix.
 
-**Validation status.** Native GF smoke 85127 completed 300 frames and scoring, but its interpolation uses later generated keyframes and is not a fair past-only autoregressive comparison. Retain that workflow as an offline diagnostic. The new causal controls have separate outputs; GPU image quality and stability remain to be established. Context budgets and method differences are recorded in `deployment/context_budget_audit.json`.
+**Geometry Forcing full causal evaluation.** The full launcher now runs the tested causal evaluator at 12 active slots (11 past frames plus one target), guidance 1, DDIM50, eta 0 and stabilization 0.02. It first generates and scores one complete 300-frame monitor loop, then automatically evaluates final100 on eight H200 workers and aggregates all 30,000 frames. The teacher-history diagnostic is optional and does not gate this run. Outputs use `geometry-forcing_ar12_cfg1_re10k_100cases_seed0_job<JOB>`; logs are in `GeometryForcing/slurm_logs/`. The native workflow described above remains available only through `deployment/smoke_geometry_forcing_1h200.sh` as an offline control.
+
+**Validation status.** Causal tasks 85128_0/1/2 completed two monitor scenes with 32 frames each. CFG1 avoided CFG4's noise collapse but still showed blur and motion lag; its full-horizon quality is not established. These monitor scenes are disjoint from final100, and the full run freezes the chosen CFG1 setting. Native smoke 85127 completed 300 frames but uses future keyframes during interpolation. Context budgets and method differences are recorded in `deployment/context_budget_audit.json`.
 
 Upstream revisions are recorded in `upstream_sources.json`. Sources retain their original licenses and notices, described in [THIRD_PARTY.md](THIRD_PARTY.md). In particular, the frozen WorldMem reference is covered by S-Lab License 1.0 and is not covered by the other projects' permissive licenses.

deployment/evaluate_geometry_forcing_8h200.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-#SBATCH --job-name=geometry-forcing-re10k-native
+#SBATCH --job-name=geometry-forcing-re10k-ar12-cfg1-full
 #SBATCH --nodes=1
 #SBATCH --ntasks-per-node=8
 #SBATCH --cpus-per-task=14
@@ -19,12 +19,11 @@ ENV_ROOT=/proj/cvl/users/x_fahkh2/envs
 REPO=$ROOT/GeometryForcing
 PYTHON=$ENV_ROOT/geometry-forcing-eval/bin/python
 METRIC_PYTHON=$ENV_ROOT/geometry-forcing-eval/bin/python
-WORLD_MEM_ROOT=$ROOT/references/worldmem
 DATA_ROOT=/proj/cvl/users/x_fahkh2/WorldMem_Repro/datasets/re10k_dfot/real-estate-10k
 MANIFEST=$ROOT/deployment/manifests/re10k_real_trajectory_h100_f399_scopes_v1.csv
 SEED=0
 JOB=$SLURM_JOB_ID
-RUN=geometry-forcing_native_re10k_100cases_seed${SEED}_job${JOB}
+RUN=geometry-forcing_ar12_cfg1_re10k_100cases_seed${SEED}_job${JOB}
 OUTPUT=$REPO/outputs/$RUN
 PREFLIGHT_OUTPUT=$REPO/outputs/${RUN}_preflight
 
@@ -46,21 +45,17 @@ cd "$REPO"
 
 "$PYTHON" "$ROOT/deployment/verify_node.py" --output "$OUTPUT"
 cp "$0" "$OUTPUT/launcher.sh"
+cp "$ROOT/deployment/geometry_causal_controls.py" "$OUTPUT/probe.py"
 
-# Exercise both keyframe batches and all interpolation windows on one complete loop.
+# Check generation and scoring across the complete loop, then continue automatically.
 srun --exclusive --exact --nodes=1 --ntasks=1 --gpus-per-task=1 --cpus-per-task=14 --cpu-bind=cores \
-    "$PYTHON" "$ROOT/shared/run_and_score.py" \
-    --method geometry-forcing \
-    --workspace-root "$ROOT" \
-    --metric-python "$METRIC_PYTHON" \
-    --worldmem-root "$WORLD_MEM_ROOT" \
-    --manifest "$MANIFEST" \
+    "$PYTHON" "$ROOT/deployment/geometry_causal_controls.py" \
+    --mode ar12_cfg1 \
     --data-root "$DATA_ROOT" \
     --output "$PREFLIGHT_OUTPUT" \
-    --scope final100 \
-    --seed "$SEED" \
+    --scope monitor8 \
     --limit 1 \
-    --max-generated 300 \
+    --frames 300 \
     --rank 0 \
     --world-size 1
 
@@ -69,24 +64,19 @@ srun --exclusive --exact --nodes=1 --ntasks=1 --gpus-per-task=1 --cpus-per-task=
     --output "$PREFLIGHT_OUTPUT" \
     --manifest "$MANIFEST" \
     --dataset re10k \
-    --scope final100 \
+    --scope monitor8 \
     --seed "$SEED" \
     --expected-frames 300 \
     --limit 1
 
 srun --ntasks=8 --gpus-per-task=1 --cpus-per-task=14 --cpu-bind=cores --kill-on-bad-exit=1 \
-    "$PYTHON" "$ROOT/shared/run_and_score.py" \
-    --method geometry-forcing \
-    --workspace-root "$ROOT" \
-    --metric-python "$METRIC_PYTHON" \
-    --worldmem-root "$WORLD_MEM_ROOT" \
-    --manifest "$MANIFEST" \
+    "$PYTHON" "$ROOT/deployment/geometry_causal_controls.py" \
+    --mode ar12_cfg1 \
     --data-root "$DATA_ROOT" \
     --output "$OUTPUT" \
     --scope final100 \
-    --seed "$SEED" \
     --limit 100 \
-    --max-generated 300 \
+    --frames 300 \
     --world-size 8
 
 srun --exclusive --exact --nodes=1 --ntasks=1 --gpus-per-task=1 --cpus-per-task=14 \
@@ -95,4 +85,5 @@ srun --exclusive --exact --nodes=1 --ntasks=1 --gpus-per-task=1 --cpus-per-task=
     --manifest "$MANIFEST" \
     --dataset re10k \
     --scope final100 \
-    --seed "$SEED"
+    --seed "$SEED" \
+    --expected-frames 300

deployment/geometry_causal_controls.py

@@ -1,4 +1,4 @@
-"""Past-only GF diagnostics; the teacher-history arm is not a benchmark result."""
+"""Causal GF evaluation and diagnostics; teacher history is diagnostic only."""
 import argparse
 import json
 import os
@@ -54,12 +54,18 @@ def main():
     parser.add_argument('--mode', choices=MODES, required=True)
     parser.add_argument('--data-root', type=Path, required=True)
     parser.add_argument('--output', type=Path, required=True)
+    parser.add_argument('--scope', choices=['monitor8', 'final100'], default='monitor8')
+    parser.add_argument('--limit', type=int, default=2)
+    parser.add_argument('--frames', type=int, default=32)
+    parser.add_argument('--rank', type=int, default=int(os.environ.get('SLURM_PROCID', 0)))
+    parser.add_argument('--world-size', type=int, default=int(os.environ.get('SLURM_NTASKS', 1)))
     parser.add_argument('--dry-run', action='store_true')
     args = parser.parse_args()
     window, scale, teacher = MODES[args.mode]
     manifest_path = ROOT / 'deployment/manifests/re10k_real_trajectory_h100_f399_scopes_v1.csv'
-    rows = select_cases(manifest_path, 'monitor8', limit=2)
-    manifest = {'mode': args.mode, 'diagnostic_only': True, 'teacher_history': teacher,
+    rows = select_cases(manifest_path, args.scope, args.rank, args.world_size, args.limit)
+    manifest = {'mode': args.mode, 'diagnostic_only': teacher or args.scope != 'final100' or args.frames != 300,
+        'teacher_history': teacher, 'scope': args.scope, 'rank': args.rank, 'world_size': args.world_size,
         'workflow': 'one target at a time; no keyframes or interpolation', 'active_slots': window,
         'past_rgb_frames': window-1, 'physical_slots': 16, 'noise_padding_slots': 16-window,
         'camera_policy': 'past and current target only; current camera repeated in noise-padding slots',
@@ -70,7 +76,7 @@ def main():
         'weights': 'released state_dict; unchanged; no EMA substitution', 'parameters': 458818051,
         'trainable_parameters': 0, 'batch_size': 1, 'optimizer': None, 'lr': None, 'lora': None,
         'activation_checkpointing': False, 'precision': 'fp16 autocast; fp32 weights/cameras',
-        'scored_timeline': [100, 132], 'master_seed': 0, 'full_loop_evaluation': False,
+        'scored_timeline': [100, 100+args.frames], 'master_seed': 0, 'full_loop_evaluation': args.frames == 300,
         'torch': torch.__version__, 'initial_noise_shape': [1, 16, 3, 256, 256], 'target_noise_slot': window-1,
         'rng_note': 'same case seeds; AR8 and AR12 have different target noise slots',
         'cases': [{'scene': r['scene_id'], 'seed': case_seed(r, 0)} for r in rows]}
@@ -80,7 +86,8 @@ def main():
     args.output = args.output.resolve()
     args.data_root = args.data_root.resolve()
     args.output.mkdir(parents=True, exist_ok=True)
-    (args.output / 'manifest.json').write_text(json.dumps(manifest, indent=2))
+    manifest_file = args.output / ('manifest.json' if args.world_size == 1 else f'manifest_rank{args.rank}.json')
+    manifest_file.write_text(json.dumps(manifest, indent=2))
     os.chdir(ROOT / 'GeometryForcing')
     from evaluation.worldmem_adapter import GeometryForcingAdapter
     from algorithms.dfot.history_guidance import HistoryGuidance
@@ -89,20 +96,21 @@ def main():
     manifest['checkpoint'] = adapter.provenance['checkpoint']
     manifest['loaded_tensors'] = adapter.provenance['loaded_backbone_tensors']
     manifest['actual_parameters'] = adapter.provenance['parameters']
-    (args.output / 'manifest.json').write_text(json.dumps(manifest, indent=2))
+    manifest_file.write_text(json.dumps(manifest, indent=2))
     for row in rows:
         case = load_re10k_case(row, args.data_root)
         seed = case_seed(row, 0)
         start = time.monotonic()
-        prediction, sampler_states, global_states = generate(adapter.model, case, window, guidance, teacher, seed)
-        directory = save_case(args.output, row, prediction, {**manifest, 'seed': seed, 'generated_frames': 32,
+        prediction, sampler_states, global_states = generate(adapter.model, case, window, guidance, teacher, seed, args.frames)
+        directory = save_case(args.output, row, prediction, {**manifest, 'seed': seed, 'generated_frames': args.frames,
             'history_frames': 100, 'source_indices': case['source_indices'].tolist(), 'generation_seconds': time.monotonic()-start})
         np.savez_compressed(directory / 'rng_states.npz', sampler=sampler_states.numpy(), global_rng=global_states.numpy())
         del prediction, case
         torch.cuda.empty_cache()
         subprocess.run([sys.executable, str(ROOT / 'shared/score_case.py'), '--case-dir', str(directory),
             '--dataset', 're10k', '--data-root', str(args.data_root), '--worldmem-root', str(ROOT / 'references/worldmem')], check=True)
-    (args.output / 'complete.json').write_text(json.dumps({'mode': args.mode, 'cases': 2, 'frames': 32}))
+    complete_file = args.output / ('complete.json' if args.world_size == 1 else f'worker_{args.rank:02d}_complete.json')
+    complete_file.write_text(json.dumps({'mode': args.mode, 'rank': args.rank, 'cases': len(rows), 'frames': args.frames}))
     print('CAUSAL_CONTROL_COMPLETE', flush=True)
 
 if __name__ == '__main__':