File size: 4,138 Bytes
6694761 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | # Fractus Emergent Properties — In-Training Surgery and Checkpoint Composition
**Updated:** 2026-08-17 00:26 UTC
These properties were **discovered in production** while correcting Fractus mid-run.
They are not marketing claims; they were forced by real bugs, resumes, merges, and continued training.
---
## 1. In-training surgery (open-heart)
**Claim:** Fractus can be modified **during training** without throwing away learned weights.
### What we did live
- Unfroze Kuramoto (removed no_grad) while runs continued from the same token offsets
- Wired load-balance into the loss mid-run
- Raised MoE gate temperature
- Switched from last-token CE to **dense full-chunk CE** (stage2)
- Added scheduled sampling
- Recalibrated LR and logging
- Fixed train/gen path mismatch (RK4 alignment, tick_chunk decode)
### What was preserved
- All previously digested tokens (exact resume manifests)
- Model state_dict tensors (same shapes)
- Multi-day progress across pod restarts
### Operational rule
1. Save / keep current .pt
2. Patch body (code / objective / routing)
3. Reload weights
4. Resume at recorded token offset
5. Document the surgery
**This is mid-training operability**, not a full retrain.
See also: docs/OPERABILITY_MIDTRAIN.md
---
## 2. Parallel brains then merge
**Claim:** You can train **many different .pt files** (shards / GPUs / runs) and **merge** them into one Fractus.
### Proven path (this run)
- 4 independent GPU runs on 4 data shards
- Each writes checkpoints/fractus_1b_gpu0.pt ... gpu3.pt
- **Mean-merge** of floating tensors -> unified checkpoint
- Example artifact: checkpoints/FRACTUS_1B_STAGE2_MERGED.pt
- Unified .pt still generates (different attractors than a single shard)
### Workflow
Constraint: **same architecture shapes** (d_model, layers, experts, etc.).
---
## 3. Compose an already-trained Fractus with more checkpoints
**Claim:** Take a Fractus already trained, merge it with other compatible .pt files, then keep training — the system **accumulates** rather than restarting from zero.
### Composition loop
### What grow means here
Two related notions:
1. **Compositional growth (proven in this run)**
- Knowledge / weights from multiple runs can be averaged into one brain
- Then training continues on the merged brain
- Digestion is not discarded
2. **Architectural growth (design of Fractus / paliers)**
- Progressive growth: width, depth, experts can increase across paliers
- maybe_grow adds capacity when routing demands it
- Documented in the Chinchilla/growth notes
Merging many same-shape .pt files is **composition**.
Growing parameter count is **structural growth**.
Both are part of the Fractus operating model; this production run proved composition + mid-train surgery under fire.
---
## 4. Why this matters
Standard large training runs treat the job as fragile:
- change the objective -> often restart
- multi-GPU shard brains -> hard to recombine
- decode bugs -> blamed on needs more train only
Fractus production forced a different picture:
| Property | Observation |
|----------|-------------|
| Operable mid-train | Yes — weights kept, body patched |
| Multi-pt parallel train | Yes — 4 shards |
| Merge to one Fractus | Yes — mean-merge works |
| Resume after surgery | Yes — exact token offsets |
| Train then merge then train | Yes — intended workflow |
---
## 5. Limits (honest)
- Merge requires **matching tensor shapes**
- Naive mean-merge of very divergent specialists can **soup** skills
- Composition is not magic AGI; it is a **workflow property** of separable brain (.pt) + body (code)
- Generation coherence still lags teacher-forced loss (see docs/LOSS_VS_GEN.md)
---
## 6. Related documents
- docs/OPERABILITY_MIDTRAIN.md — open-heart principle and stage2 surgery
- docs/DISCOVERY_LOG.md — bugs and emergent features list
- docs/MASTER_RUN_LOG.md — full chronology
- docs/2026-08-12-fractus-chinchilla.md — progressive growth / paliers
- docs/LOSS_VS_GEN.md — why TF loss is not gen quality
---
*Discovered by operating Fractus under real training pressure, not by design slide.*
|