docs/RUN_8GPU_HOURLY.md

8-GPU Relaunch + Hourly HF Sync

Updated: 2026-08-17 18:34 UTC

Context

Previous 4x5090 pod died (Docker RwLayer nil). Resumed on new 8xRTX 5090 from HF recovery weights.

Resume point

• Weight: checkpoints/FRACTUS_1B_STAGE2_MERGED.pt
• Code: fractus/ + scripts/fast4gpu_boost.py
• Data: 8 shards x 40M tokens from thefinalboss/fractus-datasets
• Torch: 2.11.0+cu128 (required for 5090)

Training config

• 8 independent GPU processes
• BATCH=2 (B=4 OOM on this stack)
• SEQ=128 LR=7e-4
• SS sequential 0.25/0.20
• compile disabled for VRAM
• all start from STAGE2_MERGED

Hourly backup

Daemon: /workspace/hourly_hf_sync.py every 3600s

1. Upload fractus_1b_gpu0.pt through gpu7.pt (8 brains)
2. Mean-merge to FRACTUS_1B_8GPU_MERGED.pt
3. Also publish as FRACTUS_1B_STAGE2_MERGED.pt recovery alias
4. HOURLY_MANIFEST.json

Policy: 8 individual + 1 unified every hour.

Related docs

• COMPOSABILITY_AND_SURGERY.md / OPERABILITY_MIDTRAIN.md
• TRUSTED_LOSS.md / LOSS_VS_GEN.md
• DECODE_SURGERY.md / GENERATE_ALIGNED.md
• BOOST_B4_RECOVERY.md
• MASTER_RUN_LOG.md