# 2026-08-27/28 night — SS live, 8×5090, HF hourly x8

All times **UTC** unless marked PDT. Pod `t8620qxxqgn6x4`, US-NC-2, 8× RTX 5090, community $5.52/h.
Code tree: `fractus-p0` (this package). Weights: `thefinalboss/fractus-cte/checkpoints/x8run/`.

## Snapshot at 03:50 UTC 2026-08-28 (~20:50 PDT 2026-08-27)

| GPU | tokens (phase-2 shard) | ema_tf | ema_ss | tok/s | VRAM |
|-----|------------------------|--------|--------|-------|------|
| 0 | 41,931,776 | 7.68 | 18.94 | 989 | 27.4 GB |
| 1 | 40,197,120 | 6.87 | 17.61 | 989 | 27.6 GB |
| 2 | 39,945,216 | 14.95 | 27.14 | 983 | 27.6 GB |
| 3 | 41,035,776 | 13.03 | 24.58 | 987 | 27.6 GB |
| 4 | 39,173,120 | 37.83 | 42.51 | 982 | 27.6 GB |
| 5 | 40,068,096 | 20.68 | 27.76 | 980 | 27.4 GB |
| 6 | 40,314,880 | 19.23 | 26.58 | 987 | 27.5 GB |
| 7 | 38,942,720 | 29.96 | 36.90 | 991 | 27.6 GB |

- Shard length: 429,896,462 tokens/GPU (phase-2 memmap). ~9.1–9.8% of this pass.
- Recipe now: **B=4 SEQ=128 SS_RATE=1.0** LR=7e-4, `FRACTUS_ATTN_IMPL=chunked`, `BLOCK_CKPT=0`.
- All 8 `_live.pt` present on disk (4.3–4.4 GB each).
- unique@40 PREFIX still **NO-GO** on last probe before SS (mean unique ≈ 3.3). SS is the intervention; speech is not claimed yet.

## Timeline

### 2026-08-27 19:05–19:20 PDT — diagnosis

- 8 trainers died mid-save: workspace volume **80 GB full**. `PytorchStreamWriter` / unexpected pos.
- unique@40 PREFIX NO-GO mean_u=3.3. Probe on `gpu0_live` (440/440 tensors, 1.05B):
  - "Hello, my name is" → HOME/Rac loop (unique 2)
  - "Once upon a time" → relation/AFTA (unique 3)
- Teacher-forced CE was learning (GPU0 tf ~5–8). **Free-run was not.** SS_RATE was 0 because of a checkpoint bug.

### 2026-08-27 19:20–19:35 PDT — SS surgery (code)

Cause of SS crash: `torch.utils.checkpoint` inside `chunked_cross_entropy` / `chunked_repeat_logprob` colliding on the **second** backward (saved logits `[N,50257]` vs recomputed hidden `[N,1280]`).

Fixes landed in-tree:

1. `fractus/train/v4_step.py`
   - `snapshot_carry` / `restore_carry` so SS replays the **same** thought + attn_S/z + Kuramoto phases as the TF chunk.
   - `v4_ss_pass` uses `tick_chunk_train` + **dense** CE (B·SEQ·vocab is tiny at B≤4). No CE-checkpoint on the SS pass.
2. `fractus/nn/ce.py` + `fractus/train/ar_loss.py`
   - skip activation checkpoint when `N <= ce_chunk` (the 1B B=2/4 regime).
3. `scripts/fast4gpu_boost_v4.py`
   - restore carry before SS; `block_ckpt=False` on SS.

First live SS lines (B=2): `tf=29.6 ss=42.2` GPU2, no CheckpointError. ~600 tok/s (two forwards).

### 2026-08-27 19:38 PDT — volume

- REST PATCH `volumeInGb` 80 → **200**. Public SSH dropped ~1 min (new port 35370).
- Trainers restarted. Disk **146 GB free**. VRAM B=2+SS ~18/32 GB.

### 2026-08-27 19:41–19:43 PDT — x8 save + HF push

- Save every `n % 4000 == (GPU * 500) % 4000` — **all 8**, staggered.
- `hf_push_x8.py` → `thefinalboss/fractus-cte/checkpoints/x8run/fractus_1b_gpu{i}.pt` + `RESUME_gpu{i}.json`.
- Interval corrected from 15 min to **3600 s**.

### 2026-08-27 20:01 PDT — batch push

- B=2 → **B=4**, SS still 1.0.
- Holds: **27.1 / 32 GB**, ~**1070 tok/s**/GPU first steps, later ~980–990.
- Aggregate ~7.9k tok/s. B=5 not taken (OOM risk).

## What this night did **not** do

- Did not finish the 3.44B phase-2 pass (~9% of current shards).
- Did not make unique@40 GO. SS is on; generation quality is the next measured gate, not a slogan.
- Did not re-ingest phase-1 (~1.5B already in the X8 weights).

## How to resume

```
CKPT_IN=checkpoints/fractus_1b_gpu{i}_live.pt
START_TOKEN=<tokens_processed from config or RESUME_gpu{i}.json>
BATCH=4 SS_RATE=1.0 BLOCK_CKPT=0 FRACTUS_ATTN_IMPL=chunked
```

Pusher: `python3 -u /workspace/hf_push_x8.py` (hourly).