Fractus Discovery Log — Bugs, Optimizations, Emergent Features
Updated: 2026-09-02 (adds sandbox live-operation discoveries on the X8 unified checkpoint)
This document records what we found while running Fractus-1B — things that go beyond the original design notes. Empirical, not marketing.
1. Critical training bugs found (and fixed)
1.1 Kuramoto was not learning during training
Symptom: omega stayed near init (±0.05), phase order parameter r ≈ 0.01–0.03, soft dynamics.
Root cause: In CTEBlock.tick_chunk_core, Kuramoto integration ran under torch.no_grad():
Comment in code even said "clock, not learned". So CE loss never reached omega / coupling.
Fix: Remove no_grad around Kuramoto; keep phase state detached for carry, keep differentiable theta for MoE routing so parameters receive gradients.
Status: Fixed in fractus/continuous_engine.py (surgery 2026-08-16).
1.2 Load-balance loss was computed then thrown away
Symptom: Offline probe showed ~70% experts dead (90+/128), only 2–3 experts active per block.
Root causes:
tick_chunk_traindidtotal_lb + lb.detach()— no gradient through LBfast4gpu.pyoptimized only cross-entropy — never addedlb_lossto the loss
Fix:
- Keep LB in the graph
tick_chunk_trainreturns(logits, lb_loss)- Surgery trainer:
loss = CE + 0.02 * lb
Status: Fixed; live lb≈14 on all GPUs after resume.
1.3 Probe false alarm: "all phases are zero"
Symptom: A probe script reported all Kuramoto phases at 0.0.
Root cause: Script called reset_thought() before measuring — which zeros state buffers.
Reality in checkpoints: phases nonzero, std ≈ 1.8 across all 16 blocks × 4 GPUs.
Lesson: Never diagnose dynamical state after an intentional reset.
2. Optimizations discovered in production
| Optimization | What we learned |
|---|---|
| 4-GPU independent shards + mean-merge | Works; unified .pt generates; different lexical attractors than single shard |
| Resume from exact token offset | Manifest-driven start_token preserves progress across pod reboot |
| Gate temperature ↑ (1.0 → 2.5) | Softens von Mises routing; more experts can enter the top-k mix |
| Omega scale ×4 on resume | Restores phase-rate diversity without wiping weights |
tick_vec multimodal path |
Vision patches can drive CTE without touching token embedding |
| CPU eyes prototype (CIFAR) | Small CTE+PatchEmbed learns real images offline while 1B trains on GPU |
3. Features that emerged beyond the original plan
3.1 Operable / open-heart model
Weights (.pt) + body (fractus/ code) are separable. We can:
- merge brains
- change routing temperature
- inject LB pressure
- add vision front-end without a full retrain from zero.
3.2 Infinite-ish checkpoint fusion (same architecture)
Compatible checkpoints can be mean-merged and trained again:
train → merge → train → merge → ...
Constraint: same shapes (d_model, layers, experts, etc.). Divergent merges can soup skills; shard-merge is the proven path.
3.3 Mid-training generation behavior
At loss ~33→25, generation pipeline works but outputs word-level repetition collapse (real tokens stuck in loops: "Colorado", "Population", "Fate", "thinks"…). Documents that Fractus emits lexical tokens before coherent sentences.
3.4 Parallel modality track
Text 1B digestion and vision prototype can run in parallel (GPU text + CPU eyes) without stopping the main run.
3.5 Routing pathology as first-class debug target
Expert-hit histograms + phase order parameter r are necessary metrics. Loss alone hides "model learns with 3 experts".
4. Live metrics after routing surgery (resume)
Resume offsets preserved from pre-crash run (~176–187M tokens/GPU).
| GPU | Resume start | Snapshot tokens | CE loss | lb |
|---|---|---|---|---|
| 0 | 176,281,600 | 176,614,400 | 36.5 | 14.026 |
| 1 | 186,137,600 | 186,444,800 | 12.4 | 14.024 |
| 2 | 186,854,400 | 187,212,800 | 34.7 | 14.026 |
| 3 | 183,833,600 | 184,166,400 | 41.0 | 14.027 |
Post-surgery CE can spike briefly then fall (routing distribution shift). GPU1 recovered fastest into the teens.
5. What this means for the Fractus thesis
Fractus is not only "another 1B trained on shards". The run forced discovery of:
- Silent non-learning of the phase clock under
no_grad - Silent expert death without LB in the loss
- Composable checkpoints as a workflow
- Operability (surgery without discarding digestion)
- A body that speaks before the brain converges — decode-level surgery on the same weights turns 2.14 unique/48 (greedy carry) into 32.86 unique/48 fluid, repeat-free speech. The collapse is a sampling-regime attractor, not a weights defect: the thesis holds above the training gate (see §7).
The architecture does more than the first README described — because production training exposed the dynamical bottlenecks.
6. Discoveries from the optimization session (2026-08-22/23)
6.1 The "attention matmul" was a disguised cumsum
The production kernel computed causal sums via
einsum("tj,bjpq->btpq", tril_mask, outer) — an O(C²·dH²) masked contraction
that IS mathematically a prefix sum. Replaced by torch.cumsum(outer, dim=1)
(O(C·dH²)), proven bit-close to the kept einsum reference (forward, gradients,
carried S/z). Lesson: profile semantics, not just FLOPs — the mask+matmul form
hid a linear-scan structure for months.
6.2 Discrete routing makes strict equivalence claims wrong at block level
topk over von Mises gates is DISCRETE: a float32 rounding difference near a gate tie flips expert choice for isolated tokens (measure-zero boundary), and depth amplifies it. Honest framing adopted: continuous stages are proven bit-close; full-block equality is asserted statistically (median at rounding scale, mismatch fraction bounded). Tests encode exactly this.
6.3 Two constructions under one seed have DIFFERENT weights
manual_seed(s); A = Model(); B = Model() gives two different networks — the
RNG stream continues across constructions. Any A/B equivalence test must clone
weights (load_state_dict) instead of re-seeding. This masqueraded as a
"routing divergence" for half a session before being identified.
6.4 CPU commit-limit segfaults are data, not noise
At 1B shapes the reference kernels materialize (G, C, dH, dH) and hit the native commit limit on a RAM-constrained host (segfault, not MemoryError). The benchmark harness now runs each cell in its own subprocess so one crash documents itself without killing the table — and the crash boundary itself measures the memory win of the chunked kernel.
7. Sandbox live operation on the X8 unified checkpoint (2026-09-02)
External CPU sandbox (fp32) loaded FRACTUS_1B_X8_MERGED.pt and ran the full
README gate protocol + a live speech operation on the continuous state.
Empirical findings:
7.1 The generation collapse is a greedy-regime attractor, not a distribution property
Symptom: greedy carry (native continuous mode) = 48 consecutive identical tokens (2.14 unique/48). Same weights, top-150 sampling @ T 1.15 with ban 20: 32.86 unique/48, zero repeats, no 2-cycles.
Root cause: von Mises soft gates (architecture) + anti-copy/LB training spread mass across the top-150 support; the argmax landscape still holds short repetition attractors. The gate's greedy unique@40 measures the most pathological regime of the model.
Status: Measured. Action: add a second birth metric — sampled unique@48 (top-k 150, T 1.15, ban 20, continuous state). Gate v2 = greedy (formal) + sampled (the "body") + state stability. See docs/X8_CIEL_OUVERT.md §5.
7.2 The continuous (S,z) state is a memory of degradation, not a memory
Symptom: in one unreset thought stream, vocabulary overlap with earlier segments: 91 % (segment 2), 100 % (segment 3). 192-token run = a ~12-token motif macro-cycle (ban 20 breaks <20 loops, not 20+ motifs).
Root cause: attn_S += outer.detach() is unbounded, no decay — the readout
q·S / q·z becomes a function of the whole history and the effective
vocabulary shrinks with state age. A multiplicative leak on (S,z)
(λ 0.99, half-life ~69 ticks) makes continuity sustainable but does not invert
the shrinkage.
Status: Measured. Action: active forgetting (selective/importance-weighted decay, or normalize S/z to code a mean). Continuity metric: overlap < 0.7 AND unique ≥ 20 over 3+ segments.
7.3 Decode-surgery phase noise is a no-op on the tick path
Symptom: r ≈ 0.0028 in every configuration; perturbing
blk.kuramoto_phases never changes outputs.
Root cause: CTEBlock.tick_single computes a fresh θ from the hidden state
each tick and feeds it to the MoE; the stored kuramoto_phases buffer is read
only by the expert-hit counter. decode_surgery.generate_with_surgery mutates
the dead buffer.
Status: Verified in code. Action: apply phase noise to the θ the MoE actually reads (one line) or delete the dead code. Kuramoto order must be raised in training (LB under gradient), not in decode.
7.4 Publish gap: the load fix is not wired into the public path
Symptom: a fresh load of the public checkpoint runs at
moe.temperature = 1.0 (code default) — the RAW config — while QuickPod runs
at 2.5 after apply_kuramoto_routing_fix. Nothing in the public code calls
the fix; memory.py docstring promises an engine.inject_memory() that does
not exist.
Status: Measured (probe RAW = 1.0 vs FIX = 2.5). Action: README line —
"apply apply_kuramoto_routing_fix at load"; implement or de-document
inject_memory.
Keep this log updated when new probes, merges, or surgeries land.