Upload docs/DISCOVERY_LOG.md with huggingface_hub

docs/DISCOVERY_LOG.md

@@ -0,0 +1,123 @@
+# Fractus Discovery Log — Bugs, Optimizations, Emergent Features
+
+**Updated:** 2026-08-16 18:28 UTC
+
+This document records what we found *while* running Fractus-1B — things that go beyond the original design notes. Empirical, not marketing.
+
+---
+
+## 1. Critical training bugs found (and fixed)
+
+### 1.1 Kuramoto was not learning during training
+
+**Symptom:** `omega` stayed near init (±0.05), phase order parameter r ≈ 0.01–0.03, soft dynamics.
+
+**Root cause:** In `CTEBlock.tick_chunk_core`, Kuramoto integration ran under `torch.no_grad()`:
+
+
+
+Comment in code even said "clock, not learned". So CE loss never reached `omega` / coupling.
+
+**Fix:** Remove `no_grad` around Kuramoto; keep phase *state* detached for carry, keep differentiable `theta` for MoE routing so parameters receive gradients.
+
+**Status:** Fixed in `fractus/continuous_engine.py` (surgery 2026-08-16).
+
+### 1.2 Load-balance loss was computed then thrown away
+
+**Symptom:** Offline probe showed ~70% experts dead (90+/128), only 2–3 experts active per block.
+
+**Root causes:**
+1. `tick_chunk_train` did `total_lb + lb.detach()` — no gradient through LB
+2. `fast4gpu.py` optimized **only** cross-entropy — never added `lb_loss` to the loss
+
+**Fix:**
+- Keep LB in the graph
+- `tick_chunk_train` returns `(logits, lb_loss)`
+- Surgery trainer: `loss = CE + 0.02 * lb`
+
+**Status:** Fixed; live `lb≈14` on all GPUs after resume.
+
+### 1.3 Probe false alarm: "all phases are zero"
+
+**Symptom:** A probe script reported all Kuramoto phases at 0.0.
+
+**Root cause:** Script called `reset_thought()` before measuring — which zeros state buffers.
+
+**Reality in checkpoints:** phases nonzero, std ≈ 1.8 across all 16 blocks × 4 GPUs.
+
+**Lesson:** Never diagnose dynamical state after an intentional reset.
+
+---
+
+## 2. Optimizations discovered in production
+
+| Optimization | What we learned |
+|--------------|-----------------|
+| 4-GPU independent shards + mean-merge | Works; unified `.pt` generates; different lexical attractors than single shard |
+| Resume from exact token offset | Manifest-driven `start_token` preserves progress across pod reboot |
+| Gate temperature ↑ (1.0 → 2.5) | Softens von Mises routing; more experts can enter the top-k mix |
+| Omega scale ×4 on resume | Restores phase-rate diversity without wiping weights |
+| `tick_vec` multimodal path | Vision patches can drive CTE without touching token embedding |
+| CPU eyes prototype (CIFAR) | Small CTE+PatchEmbed learns real images offline while 1B trains on GPU |
+
+---
+
+## 3. Features that emerged beyond the original plan
+
+### 3.1 Operable / open-heart model
+Weights (`.pt`) + body (`fractus/` code) are separable. We can:
+- merge brains
+- change routing temperature
+- inject LB pressure
+- add vision front-end
+without a full retrain from zero.
+
+### 3.2 Infinite-ish checkpoint fusion (same architecture)
+Compatible checkpoints can be mean-merged and trained again:
+
+```
+train → merge → train → merge → ...
+```
+
+Constraint: same shapes (d_model, layers, experts, etc.). Divergent merges can soup skills; shard-merge is the proven path.
+
+### 3.3 Mid-training generation behavior
+At loss ~33→25, generation pipeline works but outputs **word-level repetition collapse** (real tokens stuck in loops: "Colorado", "Population", "Fate", "thinks"…). Documents that Fractus emits lexical tokens before coherent sentences.
+
+### 3.4 Parallel modality track
+Text 1B digestion and vision prototype can run in parallel (GPU text + CPU eyes) without stopping the main run.
+
+### 3.5 Routing pathology as first-class debug target
+Expert-hit histograms + phase order parameter `r` are necessary metrics. Loss alone hides "model learns with 3 experts".
+
+---
+
+## 4. Live metrics after routing surgery (resume)
+
+Resume offsets preserved from pre-crash run (~176–187M tokens/GPU).
+
+| GPU | Resume start | Snapshot tokens | CE loss | lb |
+|-----|--------------|-----------------|---------|-----|
+| 0 | 176,281,600 | 176,614,400 | 36.5 | 14.026 |
+| 1 | 186,137,600 | 186,444,800 | 12.4 | 14.024 |
+| 2 | 186,854,400 | 187,212,800 | 34.7 | 14.026 |
+| 3 | 183,833,600 | 184,166,400 | 41.0 | 14.027 |
+
+Post-surgery CE can spike briefly then fall (routing distribution shift). GPU1 recovered fastest into the teens.
+
+---
+
+## 5. What this means for the Fractus thesis
+
+Fractus is not only "another 1B trained on shards". The run forced discovery of:
+
+1. **Silent non-learning** of the phase clock under `no_grad`
+2. **Silent expert death** without LB in the loss
+3. **Composable checkpoints** as a workflow
+4. **Operability** (surgery without discarding digestion)
+
+The architecture does more than the first README described — because production training exposed the dynamical bottlenecks.
+
+---
+
+*Keep this log updated when new probes, merges, or surgeries land.*