Upload docs/2026-08-28-NIGHT-SS-X8.md with huggingface_hub

docs/2026-08-28-NIGHT-SS-X8.md

@@ -0,0 +1,81 @@
+# 2026-08-27/28 night — SS live, 8×5090, HF hourly x8
+
+All times **UTC** unless marked PDT. Pod `t8620qxxqgn6x4`, US-NC-2, 8× RTX 5090, community $5.52/h.
+Code tree: `fractus-p0` (this package). Weights: `thefinalboss/fractus-cte/checkpoints/x8run/`.
+
+## Snapshot at 03:50 UTC 2026-08-28 (~20:50 PDT 2026-08-27)
+
+| GPU | tokens (phase-2 shard) | ema_tf | ema_ss | tok/s | VRAM |
+|-----|------------------------|--------|--------|-------|------|
+| 0 | 41,931,776 | 7.68 | 18.94 | 989 | 27.4 GB |
+| 1 | 40,197,120 | 6.87 | 17.61 | 989 | 27.6 GB |
+| 2 | 39,945,216 | 14.95 | 27.14 | 983 | 27.6 GB |
+| 3 | 41,035,776 | 13.03 | 24.58 | 987 | 27.6 GB |
+| 4 | 39,173,120 | 37.83 | 42.51 | 982 | 27.6 GB |
+| 5 | 40,068,096 | 20.68 | 27.76 | 980 | 27.4 GB |
+| 6 | 40,314,880 | 19.23 | 26.58 | 987 | 27.5 GB |
+| 7 | 38,942,720 | 29.96 | 36.90 | 991 | 27.6 GB |
+
+- Shard length: 429,896,462 tokens/GPU (phase-2 memmap). ~9.1–9.8% of this pass.
+- Recipe now: **B=4 SEQ=128 SS_RATE=1.0** LR=7e-4, `FRACTUS_ATTN_IMPL=chunked`, `BLOCK_CKPT=0`.
+- All 8 `_live.pt` present on disk (4.3–4.4 GB each).
+- unique@40 PREFIX still **NO-GO** on last probe before SS (mean unique ≈ 3.3). SS is the intervention; speech is not claimed yet.
+
+## Timeline
+
+### 2026-08-27 19:05–19:20 PDT — diagnosis
+
+- 8 trainers died mid-save: workspace volume **80 GB full**. `PytorchStreamWriter` / unexpected pos.
+- unique@40 PREFIX NO-GO mean_u=3.3. Probe on `gpu0_live` (440/440 tensors, 1.05B):
+  - "Hello, my name is" → HOME/Rac loop (unique 2)
+  - "Once upon a time" → relation/AFTA (unique 3)
+- Teacher-forced CE was learning (GPU0 tf ~5–8). **Free-run was not.** SS_RATE was 0 because of a checkpoint bug.
+
+### 2026-08-27 19:20–19:35 PDT — SS surgery (code)
+
+Cause of SS crash: `torch.utils.checkpoint` inside `chunked_cross_entropy` / `chunked_repeat_logprob` colliding on the **second** backward (saved logits `[N,50257]` vs recomputed hidden `[N,1280]`).
+
+Fixes landed in-tree:
+
+1. `fractus/train/v4_step.py`
+   - `snapshot_carry` / `restore_carry` so SS replays the **same** thought + attn_S/z + Kuramoto phases as the TF chunk.
+   - `v4_ss_pass` uses `tick_chunk_train` + **dense** CE (B·SEQ·vocab is tiny at B≤4). No CE-checkpoint on the SS pass.
+2. `fractus/nn/ce.py` + `fractus/train/ar_loss.py`
+   - skip activation checkpoint when `N <= ce_chunk` (the 1B B=2/4 regime).
+3. `scripts/fast4gpu_boost_v4.py`
+   - restore carry before SS; `block_ckpt=False` on SS.
+
+First live SS lines (B=2): `tf=29.6 ss=42.2` GPU2, no CheckpointError. ~600 tok/s (two forwards).
+
+### 2026-08-27 19:38 PDT — volume
+
+- REST PATCH `volumeInGb` 80 → **200**. Public SSH dropped ~1 min (new port 35370).
+- Trainers restarted. Disk **146 GB free**. VRAM B=2+SS ~18/32 GB.
+
+### 2026-08-27 19:41–19:43 PDT — x8 save + HF push
+
+- Save every `n % 4000 == (GPU * 500) % 4000` — **all 8**, staggered.
+- `hf_push_x8.py` → `thefinalboss/fractus-cte/checkpoints/x8run/fractus_1b_gpu{i}.pt` + `RESUME_gpu{i}.json`.
+- Interval corrected from 15 min to **3600 s**.
+
+### 2026-08-27 20:01 PDT — batch push
+
+- B=2 → **B=4**, SS still 1.0.
+- Holds: **27.1 / 32 GB**, ~**1070 tok/s**/GPU first steps, later ~980–990.
+- Aggregate ~7.9k tok/s. B=5 not taken (OOM risk).
+
+## What this night did **not** do
+
+- Did not finish the 3.44B phase-2 pass (~9% of current shards).
+- Did not make unique@40 GO. SS is on; generation quality is the next measured gate, not a slogan.
+- Did not re-ingest phase-1 (~1.5B already in the X8 weights).
+
+## How to resume
+
+```
+CKPT_IN=checkpoints/fractus_1b_gpu{i}_live.pt
+START_TOKEN=<tokens_processed from config or RESUME_gpu{i}.json>
+BATCH=4 SS_RATE=1.0 BLOCK_CKPT=0 FRACTUS_ATTN_IMPL=chunked
+```
+
+Pusher: `python3 -u /workspace/hf_push_x8.py` (hourly).