| # Fractus Discovery Log — Bugs, Optimizations, Emergent Features |
|
|
| **Updated:** 2026-08-16 18:28 UTC |
|
|
| This document records what we found *while* running Fractus-1B — things that go beyond the original design notes. Empirical, not marketing. |
|
|
| --- |
|
|
| ## 1. Critical training bugs found (and fixed) |
|
|
| ### 1.1 Kuramoto was not learning during training |
|
|
| **Symptom:** `omega` stayed near init (±0.05), phase order parameter r ≈ 0.01–0.03, soft dynamics. |
|
|
| **Root cause:** In `CTEBlock.tick_chunk_core`, Kuramoto integration ran under `torch.no_grad()`: |
|
|
|
|
|
|
| Comment in code even said "clock, not learned". So CE loss never reached `omega` / coupling. |
|
|
| **Fix:** Remove `no_grad` around Kuramoto; keep phase *state* detached for carry, keep differentiable `theta` for MoE routing so parameters receive gradients. |
|
|
| **Status:** Fixed in `fractus/continuous_engine.py` (surgery 2026-08-16). |
|
|
| ### 1.2 Load-balance loss was computed then thrown away |
|
|
| **Symptom:** Offline probe showed ~70% experts dead (90+/128), only 2–3 experts active per block. |
|
|
| **Root causes:** |
| 1. `tick_chunk_train` did `total_lb + lb.detach()` — no gradient through LB |
| 2. `fast4gpu.py` optimized **only** cross-entropy — never added `lb_loss` to the loss |
|
|
| **Fix:** |
| - Keep LB in the graph |
| - `tick_chunk_train` returns `(logits, lb_loss)` |
| - Surgery trainer: `loss = CE + 0.02 * lb` |
|
|
| **Status:** Fixed; live `lb≈14` on all GPUs after resume. |
|
|
| ### 1.3 Probe false alarm: "all phases are zero" |
|
|
| **Symptom:** A probe script reported all Kuramoto phases at 0.0. |
|
|
| **Root cause:** Script called `reset_thought()` before measuring — which zeros state buffers. |
|
|
| **Reality in checkpoints:** phases nonzero, std ≈ 1.8 across all 16 blocks × 4 GPUs. |
|
|
| **Lesson:** Never diagnose dynamical state after an intentional reset. |
|
|
| --- |
|
|
| ## 2. Optimizations discovered in production |
|
|
| | Optimization | What we learned | |
| |--------------|-----------------| |
| | 4-GPU independent shards + mean-merge | Works; unified `.pt` generates; different lexical attractors than single shard | |
| | Resume from exact token offset | Manifest-driven `start_token` preserves progress across pod reboot | |
| | Gate temperature ↑ (1.0 → 2.5) | Softens von Mises routing; more experts can enter the top-k mix | |
| | Omega scale ×4 on resume | Restores phase-rate diversity without wiping weights | |
| | `tick_vec` multimodal path | Vision patches can drive CTE without touching token embedding | |
| | CPU eyes prototype (CIFAR) | Small CTE+PatchEmbed learns real images offline while 1B trains on GPU | |
|
|
| --- |
|
|
| ## 3. Features that emerged beyond the original plan |
|
|
| ### 3.1 Operable / open-heart model |
| Weights (`.pt`) + body (`fractus/` code) are separable. We can: |
| - merge brains |
| - change routing temperature |
| - inject LB pressure |
| - add vision front-end |
| without a full retrain from zero. |
|
|
| ### 3.2 Infinite-ish checkpoint fusion (same architecture) |
| Compatible checkpoints can be mean-merged and trained again: |
|
|
| ``` |
| train → merge → train → merge → ... |
| ``` |
|
|
| Constraint: same shapes (d_model, layers, experts, etc.). Divergent merges can soup skills; shard-merge is the proven path. |
| |
| ### 3.3 Mid-training generation behavior |
| At loss ~33→25, generation pipeline works but outputs **word-level repetition collapse** (real tokens stuck in loops: "Colorado", "Population", "Fate", "thinks"…). Documents that Fractus emits lexical tokens before coherent sentences. |
| |
| ### 3.4 Parallel modality track |
| Text 1B digestion and vision prototype can run in parallel (GPU text + CPU eyes) without stopping the main run. |
| |
| ### 3.5 Routing pathology as first-class debug target |
| Expert-hit histograms + phase order parameter `r` are necessary metrics. Loss alone hides "model learns with 3 experts". |
| |
| --- |
| |
| ## 4. Live metrics after routing surgery (resume) |
| |
| Resume offsets preserved from pre-crash run (~176–187M tokens/GPU). |
| |
| | GPU | Resume start | Snapshot tokens | CE loss | lb | |
| |-----|--------------|-----------------|---------|-----| |
| | 0 | 176,281,600 | 176,614,400 | 36.5 | 14.026 | |
| | 1 | 186,137,600 | 186,444,800 | 12.4 | 14.024 | |
| | 2 | 186,854,400 | 187,212,800 | 34.7 | 14.026 | |
| | 3 | 183,833,600 | 184,166,400 | 41.0 | 14.027 | |
| |
| Post-surgery CE can spike briefly then fall (routing distribution shift). GPU1 recovered fastest into the teens. |
| |
| --- |
| |
| ## 5. What this means for the Fractus thesis |
| |
| Fractus is not only "another 1B trained on shards". The run forced discovery of: |
| |
| 1. **Silent non-learning** of the phase clock under `no_grad` |
| 2. **Silent expert death** without LB in the loss |
| 3. **Composable checkpoints** as a workflow |
| 4. **Operability** (surgery without discarding digestion) |
|
|
| The architecture does more than the first README described — because production training exposed the dynamical bottlenecks. |
|
|
| --- |
|
|
| *Keep this log updated when new probes, merges, or surgeries land.* |
|
|
