Fractus Emergent Properties — In-Training Surgery and Checkpoint Composition

Updated: 2026-08-17 00:26 UTC

These properties were discovered in production while correcting Fractus mid-run. They are not marketing claims; they were forced by real bugs, resumes, merges, and continued training.

1. In-training surgery (open-heart)

Claim: Fractus can be modified during training without throwing away learned weights.

What we did live

Unfroze Kuramoto (removed no_grad) while runs continued from the same token offsets
Wired load-balance into the loss mid-run
Raised MoE gate temperature
Switched from last-token CE to dense full-chunk CE (stage2)
Added scheduled sampling
Recalibrated LR and logging
Fixed train/gen path mismatch (RK4 alignment, tick_chunk decode)

What was preserved

All previously digested tokens (exact resume manifests)
Model state_dict tensors (same shapes)
Multi-day progress across pod restarts

Operational rule

Save / keep current .pt
Patch body (code / objective / routing)
Reload weights
Resume at recorded token offset
Document the surgery

This is mid-training operability, not a full retrain.

2. Parallel brains then merge

Claim: You can train many different .pt files (shards / GPUs / runs) and merge them into one Fractus.

Proven path (this run)

4 independent GPU runs on 4 data shards
Each writes checkpoints/fractus_1b_gpu0.pt ... gpu3.pt
Mean-merge of floating tensors -> unified checkpoint
- Example artifact: checkpoints/FRACTUS_1B_STAGE2_MERGED.pt
Unified .pt still generates (different attractors than a single shard)

Workflow

Constraint: same architecture shapes (d_model, layers, experts, etc.).

3. Compose an already-trained Fractus with more checkpoints

Claim: Take a Fractus already trained, merge it with other compatible .pt files, then keep training — the system accumulates rather than restarting from zero.

Composition loop

What grow means here

Two related notions:

Compositional growth (proven in this run)
- Knowledge / weights from multiple runs can be averaged into one brain
- Then training continues on the merged brain
- Digestion is not discarded
Architectural growth (design of Fractus / paliers)
- Progressive growth: width, depth, experts can increase across paliers
- maybe_grow adds capacity when routing demands it
- Documented in the Chinchilla/growth notes

Merging many same-shape .pt files is composition. Growing parameter count is structural growth. Both are part of the Fractus operating model; this production run proved composition + mid-train surgery under fire.

4. Why this matters

Standard large training runs treat the job as fragile:

change the objective -> often restart
multi-GPU shard brains -> hard to recombine
decode bugs -> blamed on needs more train only

Fractus production forced a different picture:

Property	Observation
Operable mid-train	Yes — weights kept, body patched
Multi-pt parallel train	Yes — 4 shards
Merge to one Fractus	Yes — mean-merge works
Resume after surgery	Yes — exact token offsets
Train then merge then train	Yes — intended workflow

5. Limits (honest)

Merge requires matching tensor shapes
Naive mean-merge of very divergent specialists can soup skills
Composition is not magic AGI; it is a workflow property of separable brain (.pt) + body (code)
Generation coherence still lags teacher-forced loss (see docs/LOSS_VS_GEN.md)

6. Related documents

docs/OPERABILITY_MIDTRAIN.md — open-heart principle and stage2 surgery
docs/DISCOVERY_LOG.md — bugs and emergent features list
docs/MASTER_RUN_LOG.md — full chronology
docs/2026-08-12-fractus-chinchilla.md — progressive growth / paliers
docs/LOSS_VS_GEN.md — why TF loss is not gen quality

Discovered by operating Fractus under real training pressure, not by design slide.