docs/DISCOVERY_LOG.md

# Fractus Discovery Log — Bugs, Optimizations, Emergent Features

**Updated:** 2026-09-02 (adds sandbox live-operation discoveries on the X8 unified checkpoint)

This document records what we found *while* running Fractus-1B — things that go beyond the original design notes. Empirical, not marketing.

---

## 1. Critical training bugs found (and fixed)

### 1.1 Kuramoto was not learning during training

**Symptom:** `omega` stayed near init (±0.05), phase order parameter r ≈ 0.01–0.03, soft dynamics.

**Root cause:** In `CTEBlock.tick_chunk_core`, Kuramoto integration ran under `torch.no_grad()`:



Comment in code even said "clock, not learned". So CE loss never reached `omega` / coupling.

**Fix:** Remove `no_grad` around Kuramoto; keep phase *state* detached for carry, keep differentiable `theta` for MoE routing so parameters receive gradients.

**Status:** Fixed in `fractus/continuous_engine.py` (surgery 2026-08-16).

### 1.2 Load-balance loss was computed then thrown away

**Symptom:** Offline probe showed ~70% experts dead (90+/128), only 2–3 experts active per block.

**Root causes:**
1. `tick_chunk_train` did `total_lb + lb.detach()` — no gradient through LB
2. `fast4gpu.py` optimized **only** cross-entropy — never added `lb_loss` to the loss

**Fix:**
- Keep LB in the graph
- `tick_chunk_train` returns `(logits, lb_loss)`
- Surgery trainer: `loss = CE + 0.02 * lb`

**Status:** Fixed; live `lb≈14` on all GPUs after resume.

### 1.3 Probe false alarm: "all phases are zero"

**Symptom:** A probe script reported all Kuramoto phases at 0.0.

**Root cause:** Script called `reset_thought()` before measuring — which zeros state buffers.

**Reality in checkpoints:** phases nonzero, std ≈ 1.8 across all 16 blocks × 4 GPUs.

**Lesson:** Never diagnose dynamical state after an intentional reset.

---

## 2. Optimizations discovered in production

| Optimization | What we learned |
|--------------|-----------------|
| 4-GPU independent shards + mean-merge | Works; unified `.pt` generates; different lexical attractors than single shard |
| Resume from exact token offset | Manifest-driven `start_token` preserves progress across pod reboot |
| Gate temperature ↑ (1.0 → 2.5) | Softens von Mises routing; more experts can enter the top-k mix |
| Omega scale ×4 on resume | Restores phase-rate diversity without wiping weights |
| `tick_vec` multimodal path | Vision patches can drive CTE without touching token embedding |
| CPU eyes prototype (CIFAR) | Small CTE+PatchEmbed learns real images offline while 1B trains on GPU |

---

## 3. Features that emerged beyond the original plan

### 3.1 Operable / open-heart model
Weights (`.pt`) + body (`fractus/` code) are separable. We can:
- merge brains
- change routing temperature
- inject LB pressure
- add vision front-end
without a full retrain from zero.

### 3.2 Infinite-ish checkpoint fusion (same architecture)
Compatible checkpoints can be mean-merged and trained again:

```
train → merge → train → merge → ...
```

Constraint: same shapes (d_model, layers, experts, etc.). Divergent merges can soup skills; shard-merge is the proven path.

### 3.3 Mid-training generation behavior
At loss ~33→25, generation pipeline works but outputs **word-level repetition collapse** (real tokens stuck in loops: "Colorado", "Population", "Fate", "thinks"…). Documents that Fractus emits lexical tokens before coherent sentences.

### 3.4 Parallel modality track
Text 1B digestion and vision prototype can run in parallel (GPU text + CPU eyes) without stopping the main run.

### 3.5 Routing pathology as first-class debug target
Expert-hit histograms + phase order parameter `r` are necessary metrics. Loss alone hides "model learns with 3 experts".

---

## 4. Live metrics after routing surgery (resume)

Resume offsets preserved from pre-crash run (~176–187M tokens/GPU).

| GPU | Resume start | Snapshot tokens | CE loss | lb |
|-----|--------------|-----------------|---------|-----|
| 0 | 176,281,600 | 176,614,400 | 36.5 | 14.026 |
| 1 | 186,137,600 | 186,444,800 | 12.4 | 14.024 |
| 2 | 186,854,400 | 187,212,800 | 34.7 | 14.026 |
| 3 | 183,833,600 | 184,166,400 | 41.0 | 14.027 |

Post-surgery CE can spike briefly then fall (routing distribution shift). GPU1 recovered fastest into the teens.

---

## 5. What this means for the Fractus thesis

Fractus is not only "another 1B trained on shards". The run forced discovery of:

1. **Silent non-learning** of the phase clock under `no_grad`
2. **Silent expert death** without LB in the loss
3. **Composable checkpoints** as a workflow
4. **Operability** (surgery without discarding digestion)
5. **A body that speaks before the brain converges** — decode-level surgery on the
   same weights turns 2.14 unique/48 (greedy carry) into 32.86 unique/48 fluid,
   repeat-free speech. The collapse is a sampling-regime attractor, not a weights
   defect: the thesis holds above the training gate (see §7).

The architecture does more than the first README described — because production training exposed the dynamical bottlenecks.

---

## 6. Discoveries from the optimization session (2026-08-22/23)

### 6.1 The "attention matmul" was a disguised cumsum

The production kernel computed causal sums via
`einsum("tj,bjpq->btpq", tril_mask, outer)` — an O(C²·dH²) masked contraction
that IS mathematically a prefix sum. Replaced by `torch.cumsum(outer, dim=1)`
(O(C·dH²)), proven bit-close to the kept einsum reference (forward, gradients,
carried S/z). Lesson: profile semantics, not just FLOPs — the mask+matmul form
hid a linear-scan structure for months.

### 6.2 Discrete routing makes strict equivalence claims wrong at block level

topk over von Mises gates is DISCRETE: a float32 rounding difference near a
gate tie flips expert choice for isolated tokens (measure-zero boundary),
and depth amplifies it. Honest framing adopted: continuous stages are proven
bit-close; full-block equality is asserted statistically (median at rounding
scale, mismatch fraction bounded). Tests encode exactly this.

### 6.3 Two constructions under one seed have DIFFERENT weights

`manual_seed(s); A = Model(); B = Model()` gives two different networks — the
RNG stream continues across constructions. Any A/B equivalence test must clone
weights (`load_state_dict`) instead of re-seeding. This masqueraded as a
"routing divergence" for half a session before being identified.

### 6.4 CPU commit-limit segfaults are data, not noise

At 1B shapes the reference kernels materialize (G, C, dH, dH) and hit the
native commit limit on a RAM-constrained host (segfault, not MemoryError).
The benchmark harness now runs each cell in its own subprocess so one crash
documents itself without killing the table — and the crash boundary itself
measures the memory win of the chunked kernel.

---

## 7. Sandbox live operation on the X8 unified checkpoint (2026-09-02)

External CPU sandbox (fp32) loaded `FRACTUS_1B_X8_MERGED.pt` and ran the full
README gate protocol + a live speech operation on the continuous state.
Empirical findings:

### 7.1 The generation collapse is a greedy-regime attractor, not a distribution property

**Symptom:** greedy carry (native continuous mode) = 48 consecutive identical
tokens (2.14 unique/48). Same weights, top-150 sampling @ T 1.15 with ban 20:
32.86 unique/48, zero repeats, no 2-cycles.

**Root cause:** von Mises soft gates (architecture) + anti-copy/LB training
spread mass across the top-150 support; the argmax landscape still holds short
repetition attractors. The gate's greedy unique@40 measures the most
pathological regime of the model.

**Status:** Measured. Action: add a second birth metric — sampled
unique@48 (top-k 150, T 1.15, ban 20, continuous state). Gate v2 = greedy
(formal) + sampled (the "body") + state stability. See
docs/X8_CIEL_OUVERT.md §5.

### 7.2 The continuous (S,z) state is a memory of degradation, not a memory

**Symptom:** in one unreset thought stream, vocabulary overlap with earlier
segments: 91 % (segment 2), 100 % (segment 3). 192-token run = a ~12-token
motif macro-cycle (ban 20 breaks <20 loops, not 20+ motifs).

**Root cause:** `attn_S += outer.detach()` is unbounded, no decay — the readout
`q·S / q·z` becomes a function of the whole history and the effective
vocabulary shrinks with state age. A multiplicative leak on (S,z)
(λ 0.99, half-life ~69 ticks) makes continuity sustainable but does not invert
the shrinkage.

**Status:** Measured. Action: active forgetting (selective/importance-weighted
decay, or normalize S/z to code a mean). Continuity metric: overlap < 0.7 AND
unique ≥ 20 over 3+ segments.

### 7.3 Decode-surgery phase noise is a no-op on the tick path

**Symptom:** r ≈ 0.0028 in every configuration; perturbing
`blk.kuramoto_phases` never changes outputs.

**Root cause:** `CTEBlock.tick_single` computes a fresh θ from the hidden state
each tick and feeds it to the MoE; the stored `kuramoto_phases` buffer is read
only by the expert-hit counter. `decode_surgery.generate_with_surgery` mutates
the dead buffer.

**Status:** Verified in code. Action: apply phase noise to the θ the MoE
actually reads (one line) or delete the dead code. Kuramoto order must be
raised in training (LB under gradient), not in decode.

### 7.4 Publish gap: the load fix is not wired into the public path

**Symptom:** a fresh load of the public checkpoint runs at
`moe.temperature = 1.0` (code default) — the RAW config — while QuickPod runs
at 2.5 after `apply_kuramoto_routing_fix`. Nothing in the public code calls
the fix; `memory.py` docstring promises an `engine.inject_memory()` that does
not exist.

**Status:** Measured (probe RAW = 1.0 vs FIX = 2.5). Action: README line —
"apply `apply_kuramoto_routing_fix` at load"; implement or de-document
`inject_memory`.

---

---

*Keep this log updated when new probes, merges, or surgeries land.*