docs/OPS_WHILE_AWAY.md

# Ops while away — 2026-08-17T21:01:43.997574+00:00

## Running now
- 8x GPU B=3 SS train on **1.52B tokenized corpus** (all .pt available on HF dataset)
- shard_len per GPU ≈ **190,125,756** tokens
- Progress tracker: `/workspace/progress_tracker.py` → `checkpoints/RESUME_MANIFEST_8GPU.json` every 2 min
- Hourly HF Xet sync of 8 ckpts + merge (daemon)

## Anti re-ingest
- Manifest stores `start_token_next` per GPU from live logs
- On any restart: use `START_TOKEN` from manifest, never 0 if progress > 0
- Phase2 job tokenizes raw jsonl/gz (cognitive_skills, neuro_code_math, paradigms, repos...) into **new** `shard_phase2_gpu*.pt` — separate stream for after phase1

## Missing 4.23B
- `full_corpus.pt` (4.23B) was **never on HF** — only referenced in scripts; lived on dead pod
- Recoverable tokenized on HF now: **~1.52B**
- Phase2 raw tokenize aims to add remaining text not fully covered by those .pt files

## When phase1 ends (~40h @ 1300 tok/s)
1. Read RESUME_MANIFEST
2. Switch SHARD to `shard_phase2_gpu{i}.pt` START_TOKEN=0
3. Or concat phase2 onto end of phase1 shards if preferred

## Do not
- Destroy pod
- Restart from START_TOKEN=0 without reading manifest