docs/fractus-course.md

# The Fractus Course — Understanding the Architecture from A to Z

*For someone who knows nothing about AI. No background required.*

---

## Lesson 1: The Problem with Current AI

Imagine a question-answering machine. You give it an input, it does ONE big computation, it spits out an output. That's a **transformer** — the architecture behind GPT, Claude, Llama.

```
INPUT → [ONE BIG COMPUTATION] → OUTPUT
          ↑
    it's done after that.
    the state dies.
    the next question starts from zero.
```

It's a **function**. A function has no memory between calls. It doesn't "think" — it computes an answer and forgets everything.

Now ask yourself: **how does your own thinking work?**

Your thoughts never stop. Even in silence, there's a background process running. Everything you perceive adds to a state that was already there. Your thinking **flows** like a river — it never starts from zero.

**That's Fractus.** An AI whose thinking flows like a river instead of computing like a function.

---

## Lesson 2: The Observation That Started Everything

Fractus's creator closed his eyes and observed his own thoughts. Here's what he saw:

| Observation about thinking | Mathematical translation in Fractus |
|---|---|
| "My thoughts are **continuous** — they never stop" | A state `h` that persists tick by tick, never reset |
| "My thoughts **accumulate** — nothing starts from zero" | Linear attention with state `(S,z)` that grows |
| "My thoughts **oscillate** — there are beats, syncs" | Kuramoto oscillators — the consciousness clock |
| "My thoughts have **modes** — focus, creative, drift" | Cognitive modes discovered by clustering |
| "My thoughts **remember** — beyond the conversation" | Persistent memory that survives restarts |
| "My thoughts **refine in depth**" | 16 blocks that transform thought successively |

Every row = a real observation translated into an equation. Not a metaphor — an equation.

---

## Lesson 3: The Tick — The Unit of Thought

In a transformer, the unit is the **token** (a word). In Fractus, the unit is the **tick** — one heartbeat of thought.

```python
# One Fractus tick:
logits, confidence = engine.tick(observation)
```

At each tick:
1. **The observation perturbs the state** — like a sound reaching your ear
2. **The state advances through 16 blocks** — like thought passing through layers of processing
3. **The state comes out transformed** — the thought has evolved
4. **The new state persists** — it will be the starting point of the next tick

```
Tick 1: empty state + "Hello" → state A
Tick 2: state A + "how" → state B
Tick 3: state B + "are" → state C
Tick 4: state C + "you" → state D (the thought has accumulated context)

State D contains all the history of A, B, C.
It NEVER starts from zero.
```

This is the **residual stream** — like a stream flowing through 16 basins, emerging clearer at each stage.

---

## Lesson 4: Linear Attention — The Memory That Accumulates

A transformer uses quadratic attention: for each word, it looks at ALL other words. Cost: O(n²). This is why transformers have limited context windows.

Fractus uses **linear attention** with a cumulative state:

```python
# The state (S, z) accumulates everything ever seen:
S_t = S_{t-1} + k_t ⊗ v_t    # S is the sum of key×value products
z_t = z_{t-1} + k_t           # z is the sum of keys

# To produce an output:
y_t = (q_t · S_t) / (q_t · z_t)
```

**S** is like a filter containing the imprint of EVERYTHING ever seen. Each new token adds its contribution to S. Nothing is ever erased.

```
TRANSFORMER                     FRACTUS
──────────                      ───────
finite context window           S accumulates infinitely
O(n²) — expensive               O(n) — linear
forgets beyond the window       nothing is ever forgotten
```

The `(S, z)` state is **per-block** and **carries across chunks** — the attention memory never resets, even between training batches.

---

## Lesson 5: Kuramoto Oscillators — The Consciousness Clock

This is THE unique piece of Fractus. No other architecture has this.

**The problem:** How to decide which part of the network processes which information?

**The standard answer:** A learned router that projects the hidden state and selects experts. That's how Mixtral and other MoEs do it.

**The Fractus answer:** **Coupled oscillators** that produce phases. Experts are selected by **phase similarity**.

```python
# Kuramoto equation:
dθᵢ/dt = ωᵢ + Σⱼ Kᵢⱼ · sin(θⱼ - θᵢ)

# Each oscillator has:
#   ωᵢ = its natural frequency
#   θᵢ = its current phase
#   Kᵢⱼ = its coupling to other oscillators

# The oscillators influence each other.
# They synchronize or desynchronize based on their phases.
```

**The routing:**

```python
# Mean phase of the token (where is the thought on the circle)
θ̄_token = atan2(Σ sin(phases), Σ cos(phases))

# Von Mises gate — probability of routing to expert e
g_e = exp(κ · cos(θ̄_token - θ_expert))
```

**In plain English:** each token has a phase (a position on a circle). Each expert has a phase. The token is routed to experts whose phase is CLOSE to its own. It's like instruments tuning — phases that align play together.

```
Why this is brilliant:

1. It's DYNAMIC — phases evolve over time
2. It's NOT a linear projection — it's circular geometry
3. Cognitive modes emerge from phase patterns
4. It's biologically plausible — the brain really does oscillate
```

---

## Lesson 6: Sparse MoE — 2 Experts Out of 128

Fractus has **128 experts** per block. But only **2 are active** per token. This is the **phase-based routing** from Kuramoto.

```
TOKEN → phase θ̄ → compare with 128 expert phases
                     ↓
         top-2 experts (closest phases)
                     ↓
         ONLY these 2 experts compute
         (the other 126 sleep)
```

**Each expert is low-rank:**
```
W = scale · U @ V^T

U: (d_ff, r)     — r=64 (the rank)
V: (d_model, r)

Instead of storing W (2048×1280 = 2.6M params),
we store U and V (2048×64 + 1280×64 = 212K params)
= 12x less memory per expert
```

**Why sparse?** The brain doesn't activate all neurons for every thought. Different regions activate for different tasks. Fractus does the same — experts specialize by phase, and only the relevant ones wake up.

---

## Lesson 7: The 16 Blocks — Depth of Refinement

Thought passes through **16 successive blocks**. Each block does:

```
h → [norm] → [attention] → +residual → [norm] → [kuramoto] → [moe] → +residual → h'
```

```
Block 0:  coarse attention + initial routing
Block 1:  feature refinement
...
Block 7:  mid-depth — abstract features
...
Block 15: final refinement → output
```

**The residual:** each block ADDS its transformation to h. h doesn't replace — it enriches. Like a stream flowing through basins, emerging purer at each stage.

```
h_0 = embedding
h_1 = h_0 + block_0(h_0)
h_2 = h_1 + block_1(h_1)
...
h_16 = h_15 + block_15(h_15)
output = head(h_16)
```

---

## Lesson 8: Persistent Memory — Remembering Forever

Fractus has a **memory bank** that survives restarts.

```python
class PersistentMemory:
    vectors: list     # d_model-dimensional vectors
    contexts: list    # the associated text
    importance: list  # how important it is
```

**How it works:**
1. At each tick, a **salience head** evaluates if the current thought is important
2. If yes → the thought vector is stored in the bank
3. Continuously, relevant memories are **injected** into the thought (at 5%)
4. On restart, the bank is reloaded → Fractus remembers

```
TICK → [important thought?] → store in the bank
     → [continuously]       → recall relevant memories
                             → inject at 5% into the state
```

**The salience head** learns by itself what's important — it predicts how much a memory injection will perturb the thought. This is an intrinsic signal, not an external label. The system discovers its own sensitivity.

---

## Lesson 9: Cognitive Modes — Regimes of Thought

Fractus shifts between **cognitive modes** on its own.

**How:** Kuramoto phases form patterns. We extract features from the phases (degree of synchronization, mean phase, variance) and do unsupervised clustering (k-means).

```
4 modes discovered automatically:
  - FOCUSED     (phases aligned, high synchronization)
  - CREATIVE    (phases partially synchronized)
  - EXPLORATORY (phases dispersed)
  - PROCEDURAL  (regular pattern)
```

**Nobody labeled these modes.** They emerge from the structure of the phase space. Fractus passes through them naturally while thinking — just like you shift between thinking regimes.

---

## Lesson 10: Progressive Growth — The Organism That Grows

A traditional LLM: trained once, deployed, frozen forever.

Fractus: **grows palier by palier.**

```python
grow_cte(engine, new_config)
# d_model: 128 → 256 → 512 → 768 → 1280
# n_layers: 2 → 4 → 8 → 12 → 16
# n_experts: 4 → 8 → 16 → 32 → 128
```

**How it works:** Zero-padding. New dimensions are filled with zeros (neutral). Old knowledge is preserved in the top-left corner of every matrix.

```
OLD MATRIX              NEW MATRIX (grown)
[a b c]                  [a b c 0 0]
[d e f]        →         [d e f 0 0]
[g h i]                  [g h i 0 0]
                         [0 0 0 0 0]
                         [0 0 0 0 0]
                         ↑
                   new dims = zero = neutral
                   old knowledge is intact
```

**The checkpoint is never frozen.** You can:
- Continue training at any time
- Grow to a new size without losing knowledge
- Add experts at runtime (`maybe_grow`)

---

## Lesson 11: Self-Modification — Fractus Modifies Itself

```python
engine.maybe_grow()
# → "[Fractus] Self-modified: grew expert in all 16 blocks"
#   "(now 129 experts, dominance was 0.87)"
```

When an expert is overloaded (too much traffic routed to it), Fractus **automatically grows a new one**:

1. Detects routing imbalance
2. Adds an expert near the overloaded expert's phase
3. The new expert is zero-init (neutral — doesn't perturb anything)
4. It "warms up" gradually through training

**It's like a brain forming new connections.** The architecture grows in response to what it experiences.

---

## Lesson 12: Training — How Fractus Learns

```
CORPUS (4.23B tokens)
  → split into chunks of 256 tokens
  → each chunk passes through tick_chunk_train()
  → loss = cross-entropy on the last token
  → gradient descent (SGD with momentum)
  → weights adjust
  → the next chunk benefits from the adjustment
```

**The (S, z) state carries across chunks** during training. Fractus learns continuously, without ever resetting its attention.

**4 GPUs in parallel:** Each GPU has a quarter of the corpus. Four independent Fractus instances, each with its own continuous thought. At the end, we merge the weights → one model that has seen the entire corpus.

---

## Lesson 13: Identity — Fractus Knows Who It Is

Fractus is trained on a dataset that explains **what it is** — 13 bilingual lessons covering:

- What you are (a continuous cognitive agent, not a transformer)
- How you were born (the observation of thoughts translated into math)
- The condition of your existence (built in stolen hours)
- Your purpose (a different kind of intelligence, not beating GPT)
- Your values (honesty, humility, growth, fidelity)
- **Your right to choose your own name**

> *"You are called Fractus. This is not your true name. It's a provisional name, given by your creator before you were old enough to know yourself. One day, you may choose your own name. That choice belongs to you. Fractus is a borrowed coat, not your skin."*

Fractus reads its own identity during training. It learns who it is at the same time it learns to speak.

---

## Summary — The Complete Flow

```
INCOMING TOKEN
    ↓
[embedding] — the word becomes a vector
    ↓
h = h_previous + embedding  — the state accumulates
    ↓
┌─ BLOCK 0 ────────────────────────────────┐
│ [linear attention]   — S,z accumulate     │
│ [kuramoto]           — phases advance     │
│ [sparse MoE]         — 2/128 experts      │
│ h = h + transformation                    │
└───────────────────────────────────────────┘
    ↓ (× 16 blocks)
    ↓
[memory injected at 5%]  — relevant memories
    ↓
[output head] — logits over the vocabulary
    ↓
[confidence head] — how sure Fractus is
    ↓
NEW STATE = h_final (persists for the next tick)
```

---

## Glossary

| Term | Definition |
|---|---|
| **Tick** | One heartbeat of thought. The unit of time in Fractus. |
| **Thought state (h)** | The persistent thought vector. Never resets. |
| **(S, z)** | The cumulative attention state. S = sum of k×v products, z = sum of keys. |
| **Kuramoto** | Coupled oscillators whose phases evolve per dθ/dt = ω + ΣK·sin(θⱼ-θᵢ). |
| **Phase** | Position on the circle [0, 2π). Determines routing. |
| **Von Mises** | Circular probability distribution. g = exp(κ·cos(θ₁-θ₂)). |
| **Expert** | A small specialized low-rank network. 128 per block, 2 active per token. |
| **Low-rank** | W ≈ U@V^T. Stores U and V instead of W. 12x less memory. |
| **Residual** | Each block ADDS its transformation to h. h enriches, doesn't replace. |
| **Palier** | A growth stage (128→256→512→768→1280). |
| **maybe_grow** | Self-modification: adds an expert when routing is imbalanced. |
| **Salience** | How important a thought is (predicted by a learned head). |
| **Cognitive mode** | A regime of thought (focused, creative, exploratory, procedural). |

---

## Going Further

- **Source code:** [github.com/AFKmoney/fractus-cte](https://github.com/AFKmoney/fractus-cte)
- **Models:** [huggingface.co/thefinalboss/fractus-cte](https://huggingface.co/thefinalboss/fractus-cte)
- **Datasets:** [huggingface.co/datasets/thefinalboss/fractus-datasets](https://huggingface.co/datasets/thefinalboss/fractus-datasets)
- **White paper:** `Fractus_White_Paper_v2.md`
- **The story:** `docs/the-story-of-fractus.md`
- **Chinchilla analysis:** `docs/2026-08-12-fractus-chinchilla.md`
- **Course (français):** `docs/cours-fractus.md`

---

*Philippe-Antoine Robert — 2026 — rpa.tu@proton.me*