docs/MASTER_RUN_LOG.md

Fractus-1B Production Run Log (Complete)

Updated: 2026-08-16 23:53 UTC

English master log of the Fractus continuous-thought 1B training run: discoveries, bugs, surgeries, metrics, and generation probes.

---

1. Architecture

Fractus is not a standard decoder-only transformer training loop.

• ContinuousThoughtEngine (CTE): residual thought state across ticks
• Per-block linear attention with continuous carry (S, z)
• Kuramoto phase oscillators for routing dynamics
• PhaseRoutedMoE (von Mises gates, top-k experts, load-balance loss)
• Tied embedding / output head
• Target: d_model=1280, 16 layers, 128 experts, top-k=2

Checkpoints store weights + dynamic state; package fractus/ is the body.

---

2. Hardware and data

• 4x NVIDIA RTX 5090
• Independent shard per GPU (~1.057B tokens each, GPT-2 BPE)
• Batch B=2, seq_len=128 (256 tokens/step)
• Optimizer: SGD momentum 0.9
• Precision: bfloat16 autocast

---

3. Chronology of interventions

Phase A — Initial distributed training

• Four independent GPU runs
• Early objective: last-position CE only
• Stable throughput, descending loss
• Generation collapsed to single-token loops

Phase B — Routing surgery (weights kept)

Bugs found:

1. Kuramoto under torch.no_grad in tick_chunk_core (omega never trained)
2. lb_loss detached and never in training loss (dead experts)
3. Hard gate temperature

Fixes: Kuramoto gradients, CE+0.02*lb, gate temp 2.5, omega diversity, exact token resume

Phase C — Stage2 dense CE

• tick_chunk_train returns logits for all positions
• Dense next-token CE over full chunk
• CE dropped hard (GPU1 into ~2 range)

Phase D — Decode surgery (weights unchanged)

• Phase/thought noise, frequency penalty, cycle bans, forced escape tokens
• Broke single-token decode lock
• Did not produce coherent English by itself

Phase E — Train/gen path mismatch

• Train: tick_chunk_core (causal attn + RK4 Kuramoto)
• Default gen: tick_single (simpler attn + Euler Kuramoto)
• Fix: align tick_single to RK4; prefer 100% tick_chunk generation
• Module: fractus/generate_aligned.py (generate_chunk, generate_window)

Phase F — Fine phase loss recalibration

• Cumulative average CE misleading near 2.0
• Batch CE + EMA; real session tok/s
• LR 1e-3 -> 5e-4

Phase G — Scheduled sampling (loss vs gen gap)

• Warm TF CE ~1.3-1.5 can coexist with broken free-run generation
• Free-run AR CE vs ground truth is catastrophic (exposure bias)
• Fix: sequential scheduled sampling
  • Pass A: teacher-forced CE + LB
  • Pass B (~30% steps): mix ~20% model samples into inputs
• Logs: tf/ema_tf and ss/ema_ss

---

4. Live SS metrics (at doc time)

• GPU 0: GPU 0: 190,809,600 tf=7.760 ema_tf=7.689 lb=14.027 670 tok/s [ss]
• GPU 1: GPU 1: 200,499,200 tf=1.292 ema_tf=1.334 lb=14.027 670 tok/s [ss]
• GPU 2: GPU 2: 201,446,400 tf=1.766 ema_tf=1.789 lb=14.026 656 tok/s [ss]
• GPU 3: GPU 3: 198,656,000 tf=5.662 ema_tf=5.998 lb=14.027 670 tok/s [ss]

---

5. Generation summary

Stage	Behavior
Early mid-train	Single-token loops
After decode surgery	Multi-token diversity, non-sentences
After path alignment	WINDOW more diverse than CHUNK
Current	Lexical noise / short cycles; not coherent prose

Expected while SS is still teaching free-run and only a fraction of each shard is consumed.

---

6. Operability principle

1. Keep .pt weights
2. Patch body (engine / trainer / decode)
3. Resume at recorded token offset
4. Document the surgery

Do not discard multi-day digestion for routing, objective, or decode bugs.

---

7. Key files

• fractus/continuous_engine.py — CTE body
• fractus/generate_aligned.py — train-aligned decode
• fractus/decode_surgery.py — optional decode stack
• scripts/fast4gpu_stage2_ss.py — current trainer
• scripts/fast4gpu_stage2_fine.py — fine phase trainer
• RESUME_MANIFEST_*.json — exact offsets
• checkpoints/fractus_1b_gpu0-3.pt — per-GPU brains

---

8. Related docs

• docs/COMPOSABILITY_AND_SURGERY.md — in-training surgery + multi-pt merge/grow

• docs/DISCOVERY_LOG.md

• docs/OPERABILITY_MIDTRAIN.md

• docs/DECODE_SURGERY.md

• docs/TRAIN_GEN_MISMATCH.md

• docs/GENERATE_ALIGNED.md

• docs/FINE_PHASE.md

• docs/LOSS_VS_GEN.md

• docs/TRUSTED_LOSS.md — which loss to trust for train vs gen

• docs/STAGE2_SS_GEN_PROBE.md (generation probe with outputs)

---

9. Conclusion

1. Learning is real under teacher-forced dense CE.
2. Generation coherence is not yet achieved.
3. Low TF loss does not imply clean free-run text.
4. Active remediation: scheduled sampling + aligned tick_chunk decode.
5. Keep rolling; re-probe generation after more SS tokens.