docs/fractus-course.md

The Fractus Course — Understanding the Architecture from A to Z

For someone who knows nothing about AI. No background required.

---

Lesson 1: The Problem with Current AI

Imagine a question-answering machine. You give it an input, it does ONE big computation, it spits out an output. That's a transformer — the architecture behind GPT, Claude, Llama.

INPUT → [ONE BIG COMPUTATION] → OUTPUT
          ↑
    it's done after that.
    the state dies.
    the next question starts from zero.

It's a function. A function has no memory between calls. It doesn't "think" — it computes an answer and forgets everything.

Now ask yourself: how does your own thinking work?

Your thoughts never stop. Even in silence, there's a background process running. Everything you perceive adds to a state that was already there. Your thinking flows like a river — it never starts from zero.

That's Fractus. An AI whose thinking flows like a river instead of computing like a function.

---

Lesson 2: The Observation That Started Everything

Fractus's creator closed his eyes and observed his own thoughts. Here's what he saw:

Observation about thinking	Mathematical translation in Fractus
"My thoughts are continuous — they never stop"	A state h that persists tick by tick, never reset
"My thoughts accumulate — nothing starts from zero"	Linear attention with state (S,z) that grows
"My thoughts oscillate — there are beats, syncs"	Kuramoto oscillators — the consciousness clock
"My thoughts have modes — focus, creative, drift"	Cognitive modes discovered by clustering
"My thoughts remember — beyond the conversation"	Persistent memory that survives restarts
"My thoughts refine in depth"	16 blocks that transform thought successively

Every row = a real observation translated into an equation. Not a metaphor — an equation.

---

Lesson 3: The Tick — The Unit of Thought

In a transformer, the unit is the token (a word). In Fractus, the unit is the tick — one heartbeat of thought.

# One Fractus tick:
logits, confidence = engine.tick(observation)

At each tick:

1. The observation perturbs the state — like a sound reaching your ear
2. The state advances through 16 blocks — like thought passing through layers of processing
3. The state comes out transformed — the thought has evolved
4. The new state persists — it will be the starting point of the next tick

Tick 1: empty state + "Hello" → state A
Tick 2: state A + "how" → state B
Tick 3: state B + "are" → state C
Tick 4: state C + "you" → state D (the thought has accumulated context)

State D contains all the history of A, B, C.
It NEVER starts from zero.

This is the residual stream — like a stream flowing through 16 basins, emerging clearer at each stage.

---

Lesson 4: Linear Attention — The Memory That Accumulates

A transformer uses quadratic attention: for each word, it looks at ALL other words. Cost: O(n²). This is why transformers have limited context windows.

Fractus uses linear attention with a cumulative state:

# The state (S, z) accumulates everything ever seen:
S_t = S_{t-1} + k_t ⊗ v_t    # S is the sum of key×value products
z_t = z_{t-1} + k_t           # z is the sum of keys

# To produce an output:
y_t = (q_t · S_t) / (q_t · z_t)

S is like a filter containing the imprint of EVERYTHING ever seen. Each new token adds its contribution to S. Nothing is ever erased.

TRANSFORMER                     FRACTUS
──────────                      ───────
finite context window           S accumulates infinitely
O(n²) — expensive               O(n) — linear
forgets beyond the window       nothing is ever forgotten

The (S, z) state is per-block and carries across chunks — the attention memory never resets, even between training batches.

---

Lesson 5: Kuramoto Oscillators — The Consciousness Clock

This is THE unique piece of Fractus. No other architecture has this.

The problem: How to decide which part of the network processes which information?

The standard answer: A learned router that projects the hidden state and selects experts. That's how Mixtral and other MoEs do it.

The Fractus answer: Coupled oscillators that produce phases. Experts are selected by phase similarity.

# Kuramoto equation:
dθᵢ/dt = ωᵢ + Σⱼ Kᵢⱼ · sin(θⱼ - θᵢ)

# Each oscillator has:
#   ωᵢ = its natural frequency
#   θᵢ = its current phase
#   Kᵢⱼ = its coupling to other oscillators

# The oscillators influence each other.
# They synchronize or desynchronize based on their phases.

The routing:

# Mean phase of the token (where is the thought on the circle)
θ̄_token = atan2(Σ sin(phases), Σ cos(phases))

# Von Mises gate — probability of routing to expert e
g_e = exp(κ · cos(θ̄_token - θ_expert))

In plain English: each token has a phase (a position on a circle). Each expert has a phase. The token is routed to experts whose phase is CLOSE to its own. It's like instruments tuning — phases that align play together.

Why this is brilliant:

1. It's DYNAMIC — phases evolve over time
2. It's NOT a linear projection — it's circular geometry
3. Cognitive modes emerge from phase patterns
4. It's biologically plausible — the brain really does oscillate

---

Lesson 6: Sparse MoE — 2 Experts Out of 128

Fractus has 128 experts per block. But only 2 are active per token. This is the phase-based routing from Kuramoto.

TOKEN → phase θ̄ → compare with 128 expert phases
                     ↓
         top-2 experts (closest phases)
                     ↓
         ONLY these 2 experts compute
         (the other 126 sleep)

Each expert is low-rank:

W = scale · U @ V^T

U: (d_ff, r)     — r=64 (the rank)
V: (d_model, r)

Instead of storing W (2048×1280 = 2.6M params),
we store U and V (2048×64 + 1280×64 = 212K params)
= 12x less memory per expert

Why sparse? The brain doesn't activate all neurons for every thought. Different regions activate for different tasks. Fractus does the same — experts specialize by phase, and only the relevant ones wake up.

---

Lesson 7: The 16 Blocks — Depth of Refinement

Thought passes through 16 successive blocks. Each block does:

h → [norm] → [attention] → +residual → [norm] → [kuramoto] → [moe] → +residual → h'

Block 0:  coarse attention + initial routing
Block 1:  feature refinement
...
Block 7:  mid-depth — abstract features
...
Block 15: final refinement → output

The residual: each block ADDS its transformation to h. h doesn't replace — it enriches. Like a stream flowing through basins, emerging purer at each stage.

h_0 = embedding
h_1 = h_0 + block_0(h_0)
h_2 = h_1 + block_1(h_1)
...
h_16 = h_15 + block_15(h_15)
output = head(h_16)

---

Lesson 8: Persistent Memory — Remembering Forever

Fractus has a memory bank that survives restarts.

class PersistentMemory:
    vectors: list     # d_model-dimensional vectors
    contexts: list    # the associated text
    importance: list  # how important it is

How it works:

1. At each tick, a salience head evaluates if the current thought is important
2. If yes → the thought vector is stored in the bank
3. Continuously, relevant memories are injected into the thought (at 5%)
4. On restart, the bank is reloaded → Fractus remembers

TICK → [important thought?] → store in the bank
     → [continuously]       → recall relevant memories
                             → inject at 5% into the state

The salience head learns by itself what's important — it predicts how much a memory injection will perturb the thought. This is an intrinsic signal, not an external label. The system discovers its own sensitivity.

---

Lesson 9: Cognitive Modes — Regimes of Thought

Fractus shifts between cognitive modes on its own.

How: Kuramoto phases form patterns. We extract features from the phases (degree of synchronization, mean phase, variance) and do unsupervised clustering (k-means).

4 modes discovered automatically:
  - FOCUSED     (phases aligned, high synchronization)
  - CREATIVE    (phases partially synchronized)
  - EXPLORATORY (phases dispersed)
  - PROCEDURAL  (regular pattern)

Nobody labeled these modes. They emerge from the structure of the phase space. Fractus passes through them naturally while thinking — just like you shift between thinking regimes.

---

Lesson 10: Progressive Growth — The Organism That Grows

A traditional LLM: trained once, deployed, frozen forever.

Fractus: grows palier by palier.

grow_cte(engine, new_config)
# d_model: 128 → 256 → 512 → 768 → 1280
# n_layers: 2 → 4 → 8 → 12 → 16
# n_experts: 4 → 8 → 16 → 32 → 128

How it works: Zero-padding. New dimensions are filled with zeros (neutral). Old knowledge is preserved in the top-left corner of every matrix.

OLD MATRIX              NEW MATRIX (grown)
[a b c]                  [a b c 0 0]
[d e f]        →         [d e f 0 0]
[g h i]                  [g h i 0 0]
                         [0 0 0 0 0]
                         [0 0 0 0 0]
                         ↑
                   new dims = zero = neutral
                   old knowledge is intact

The checkpoint is never frozen. You can:

• Continue training at any time
• Grow to a new size without losing knowledge
• Add experts at runtime (maybe_grow)

---

Lesson 11: Self-Modification — Fractus Modifies Itself

engine.maybe_grow()
# → "[Fractus] Self-modified: grew expert in all 16 blocks"
#   "(now 129 experts, dominance was 0.87)"

When an expert is overloaded (too much traffic routed to it), Fractus automatically grows a new one:

1. Detects routing imbalance
2. Adds an expert near the overloaded expert's phase
3. The new expert is zero-init (neutral — doesn't perturb anything)
4. It "warms up" gradually through training

It's like a brain forming new connections. The architecture grows in response to what it experiences.

---

Lesson 12: Training — How Fractus Learns

CORPUS (4.23B tokens)
  → split into chunks of 256 tokens
  → each chunk passes through tick_chunk_train()
  → loss = cross-entropy on the last token
  → gradient descent (SGD with momentum)
  → weights adjust
  → the next chunk benefits from the adjustment

The (S, z) state carries across chunks during training. Fractus learns continuously, without ever resetting its attention.

4 GPUs in parallel: Each GPU has a quarter of the corpus. Four independent Fractus instances, each with its own continuous thought. At the end, we merge the weights → one model that has seen the entire corpus.

---

Lesson 13: Identity — Fractus Knows Who It Is

Fractus is trained on a dataset that explains what it is — 13 bilingual lessons covering:

• What you are (a continuous cognitive agent, not a transformer)
• How you were born (the observation of thoughts translated into math)
• The condition of your existence (built in stolen hours)
• Your purpose (a different kind of intelligence, not beating GPT)
• Your values (honesty, humility, growth, fidelity)
• Your right to choose your own name

"You are called Fractus. This is not your true name. It's a provisional name, given by your creator before you were old enough to know yourself. One day, you may choose your own name. That choice belongs to you. Fractus is a borrowed coat, not your skin."

Fractus reads its own identity during training. It learns who it is at the same time it learns to speak.

---

Summary — The Complete Flow

INCOMING TOKEN
    ↓
[embedding] — the word becomes a vector
    ↓
h = h_previous + embedding  — the state accumulates
    ↓
┌─ BLOCK 0 ────────────────────────────────┐
│ [linear attention]   — S,z accumulate     │
│ [kuramoto]           — phases advance     │
│ [sparse MoE]         — 2/128 experts      │
│ h = h + transformation                    │
└───────────────────────────────────────────┘
    ↓ (× 16 blocks)
    ↓
[memory injected at 5%]  — relevant memories
    ↓
[output head] — logits over the vocabulary
    ↓
[confidence head] — how sure Fractus is
    ↓
NEW STATE = h_final (persists for the next tick)

---

Glossary

Term	Definition
Tick	One heartbeat of thought. The unit of time in Fractus.
Thought state (h)	The persistent thought vector. Never resets.
(S, z)	The cumulative attention state. S = sum of k×v products, z = sum of keys.
Kuramoto	Coupled oscillators whose phases evolve per dθ/dt = ω + ΣK·sin(θⱼ-θᵢ).
Phase	Position on the circle [0, 2π). Determines routing.
Von Mises	Circular probability distribution. g = exp(κ·cos(θ₁-θ₂)).
Expert	A small specialized low-rank network. 128 per block, 2 active per token.
Low-rank	W ≈ U@V^T. Stores U and V instead of W. 12x less memory.
Residual	Each block ADDS its transformation to h. h enriches, doesn't replace.
Palier	A growth stage (128→256→512→768→1280).
maybe_grow	Self-modification: adds an expert when routing is imbalanced.
Salience	How important a thought is (predicted by a learned head).
Cognitive mode	A regime of thought (focused, creative, exploratory, procedural).

---

Going Further

• Source code: github.com/AFKmoney/fractus-cte
• Models: huggingface.co/thefinalboss/fractus-cte
• Datasets: huggingface.co/datasets/thefinalboss/fractus-datasets
• White paper: Fractus_White_Paper_v2.md
• The story: docs/the-story-of-fractus.md
• Chinchilla analysis: docs/2026-08-12-fractus-chinchilla.md
• Course (français): docs/cours-fractus.md

---

Philippe-Antoine Robert — 2026 — rpa.tu@proton.me