| # 8-GPU Relaunch + Hourly HF Sync | |
| Updated: 2026-08-17 18:34 UTC | |
| ## Context | |
| Previous 4x5090 pod died (Docker RwLayer nil). Resumed on new 8xRTX 5090 from HF recovery weights. | |
| ## Resume point | |
| - Weight: checkpoints/FRACTUS_1B_STAGE2_MERGED.pt | |
| - Code: fractus/ + scripts/fast4gpu_boost.py | |
| - Data: 8 shards x 40M tokens from thefinalboss/fractus-datasets | |
| - Torch: 2.11.0+cu128 (required for 5090) | |
| ## Training config | |
| - 8 independent GPU processes | |
| - BATCH=2 (B=4 OOM on this stack) | |
| - SEQ=128 LR=7e-4 | |
| - SS sequential 0.25/0.20 | |
| - compile disabled for VRAM | |
| - all start from STAGE2_MERGED | |
| ## Hourly backup | |
| Daemon: /workspace/hourly_hf_sync.py every 3600s | |
| 1. Upload fractus_1b_gpu0.pt through gpu7.pt (8 brains) | |
| 2. Mean-merge to FRACTUS_1B_8GPU_MERGED.pt | |
| 3. Also publish as FRACTUS_1B_STAGE2_MERGED.pt recovery alias | |
| 4. HOURLY_MANIFEST.json | |
| Policy: 8 individual + 1 unified every hour. | |
| ## Related docs | |
| - COMPOSABILITY_AND_SURGERY.md / OPERABILITY_MIDTRAIN.md | |
| - TRUSTED_LOSS.md / LOSS_VS_GEN.md | |
| - DECODE_SURGERY.md / GENERATE_ALIGNED.md | |
| - BOOST_B4_RECOVERY.md | |
| - MASTER_RUN_LOG.md | |