| # Boost B=4 Recovery Recipe (HF) |
|
|
| **Updated:** 2026-08-17 03:10 UTC |
|
|
| ## Purpose |
|
|
| If the QuickPod volume is lost, resume training from Hugging Face using: |
|
|
| - weights: `checkpoints/FRACTUS_1B_STAGE2_MERGED.pt` (last merge uploaded **2026-08-16 19:10 UTC / 12:10 PM PDT**) |
| - trainer: `scripts/fast4gpu_boost.py` |
| - code: `fractus/` package on this repo |
|
|
| This is the **boost configuration that was running when the pod died** (B=4, ~940–960 tok/s/GPU on 5090). |
|
|
| ## Boost settings (canonical) |
|
|
| | Knob | Value | |
| |------|--------| |
| | BATCH | **4** | |
| | SEQ | 128 | |
| | LR | **7e-4** | |
| | momentum SGD | 0.9 | |
| | torch.compile | reduce-overhead | |
| | TF32 | on | |
| | dense CE | all positions via `tick_chunk_train` | |
| | LB coef | 0.02 | |
| | gate temperature | 2.5 | |
| | SS_RATE | 0.25 (fraction of steps with second pass) | |
| | SS_PROB | 0.20 (fraction of positions replaced in SS pass) | |
| | SS implementation | **sequential** (no dual graph — avoids OOM) | |
|
|
| ## On a new machine |
|
|
| ```bash |
| # 1. clone / download repo |
| huggingface-cli download thefinalboss/fractus-cte --local-dir ./fractus-cte |
| cd fractus-cte |
| |
| # 2. ensure data shards exist (re-tokenize if needed) |
| # data/shard_gpu0.pt ... shard_gpu3.pt (int64 1D token streams) |
| |
| # 3. smoke load merge (CPU ok) |
| python scripts/smoke_load_merged.py checkpoints/FRACTUS_1B_STAGE2_MERGED.pt |
| |
| # 4. launch 4 GPU boost from merge |
| CUDA_VISIBLE_DEVICES=0 GPU_ID=0 CKPT_IN=checkpoints/FRACTUS_1B_STAGE2_MERGED.pt \ |
| python -u scripts/fast4gpu_boost.py |
| |
| CUDA_VISIBLE_DEVICES=1 GPU_ID=1 CKPT_IN=checkpoints/FRACTUS_1B_STAGE2_MERGED.pt \ |
| python -u scripts/fast4gpu_boost.py |
| |
| # same for GPU 2 and 3 |
| ``` |
|
|
| Optional: `START_TOKEN=0` or a known offset if you recovered manifests. |
|
|
| ## After each save — push to HF |
|
|
| ```bash |
| huggingface-cli upload thefinalboss/fractus-cte checkpoints/fractus_1b_gpu0.pt checkpoints/fractus_1b_gpu0.pt |
| # repeat gpu1-3; mean-merge periodically |
| ``` |
|
|
| **Lesson from outage:** weights must leave the rental host continuously. |
|
|
| ## What this does / does not recover |
|
|
| | Recovered from HF merge | Not recovered (pod-only unless volume returns) | |
| |-------------------------|--------------------------------------------------| |
| | Stage2 dense-CE digestion up to merge time | Last ~hours of boost after merge | |
| | Architecture + operable train path | Exact 203M token offsets on GPU1 | |
| | Boost recipe identical to production | Live thought-state of the crashed run | |
|
|
| You do **not** start from random. You start from **stage2 merged brain** + boost loop. |
|
|
| ## Related |
|
|
| - `docs/TRUSTED_LOSS.md` — which loss to trust |
| - `docs/LOSS_VS_GEN.md` — TF vs free-run |
| - `docs/COMPOSABILITY_AND_SURGERY.md` — merge / surgery |
| - `docs/MASTER_RUN_LOG.md` — full chronology |
|
|
