---
license: mit
language:
- en
- fr
tags:
- continuous-thought
- linear-attention
- kuramoto
- moe
- rnn
library_name: pytorch
---

# Fractus-CTE

**Continuous Thought Engine.** A dynamical system with a fixed-size recurrent state, not a transformer.

Fractus belongs to the linear-attention RNN family (Katharopoulos 2020, RetNet, RWKV, GLA, DeltaNet, Mamba/S6) **plus** phase-routed MoE and a persistent thought state across chunks. Depth is time. The checkpoint is a living state (weights + phases + attention memory), not a frozen function.

| | Transformer | Fractus |
|---|---|---|
| Computation | one forward per prompt | continuous ticks / chunks |
| Memory | KV cache grows with length | fixed-size \(S, z\), thought state |
| Routing | (optional) learned softmax MoE | Kuramoto phases on a circle |
| Knowledge after train | fine-tune / RAG | **Vorax** organs, append-only `.kn` |
| Operability | replace the blob | open-heart: code changes, `.pt` shapes stay |

**Author:** Philippe-Antoine Robert ([thefinalboss](https://huggingface.co/thefinalboss))  
**License:** MIT  
**Sibling repos:** [fractus-p0](https://huggingface.co/thefinalboss/fractus-p0) (routing + v4 + DiffusionBlocks body) · [fractus-vorax](https://huggingface.co/thefinalboss/fractus-vorax) (sealed brain + ingest) · [fractus-datasets](https://huggingface.co/datasets/thefinalboss/fractus-datasets)

Full chronology: [`docs/EVOLUTION.md`](docs/EVOLUTION.md) · français [`docs/EVOLUTION.fr.md`](docs/EVOLUTION.fr.md)

---

## Architecture (1B production config)

| | |
|---|---|
| Params | ~1.165B |
| `d_model` | 1280 |
| `n_layers` | 16 CTEBlocks |
| Attention | causal **linear** (cumsum / chunked). Env `FRACTUS_ATTN_IMPL` |
| Oscillators / block | 16, coupling rank 8 |
| Experts / block | 128, top-2, phase-gated |
| Vocab | 50257 GPT-2 BPE |
| Train objective | dense next-token CE on the chunk + Switch load-balance (+ optional SS / anti-repeat in v4) |

Each block: attention → Kuramoto → PhaseRoutedMoE. \(S,z\) and phases are **per-block** and carried across chunk boundaries.

---

## What is true right now (2026-08-27)

**Done**
- Multi-GPU data-parallel shards, exact `START_TOKEN` resume, hourly mean-merge of same-shape `.pt`
- Open-heart speed work (22–26 Aug): chunked linear attention, `chunked_cross_entropy`, `BLOCK_CKPT`, atomic ckpts — **shapes unchanged**
- Phase-2 corpus on HF; freeze then x8 resume (~35M tokens/GPU on ~430M-token shards at last pushed manifest)
- P0 routing body (atan2 encode, phase carry, per-token MoE, Switch LB) in [fractus-p0](https://huggingface.co/thefinalboss/fractus-p0)
- PREFIX vs CARRY decode hole **named and measured** on a CPU mini
- Fractus-native DiffusionBlocks prototype (one CTEBlock / step, 0 new params) — experiment, not default

**Open (do not skip)**
- Attention state \(S\) still looks like \(S_t = S_{t-1} + k\otimes v\) — **no learned decay** yet (RWKV/RetNet/Mamba all added one)
- Mean-merge of independent MoE shards ≠ DDP; expert #k is not aligned across GPUs
- Kuramoto decision is still a **circle** (need \(\mathrm{MI}(\bar\theta, \text{token})\) vs tick)
- No published **matched-compute PPL** vs a vanilla transformer on a clean held-out
- CARRY length-1 generation is not the trained graph (PREFIX is the language gate)
- Eval must be split by **source**, not by line (identity text in the corpus)

Default next run: **v4 + P0 body + resume offsets + 24GB+ identical GPUs (x4 is enough)**.  
DiffusionBlocks and DDP are options. Finishing the pass without a baseline PPL is not a paper.

---

## Load

```python
from fractus.continuous_engine import ContinuousThoughtEngine
eng = ContinuousThoughtEngine.from_pretrained("checkpoints/FRACTUS_1B_PHASE2_FROZEN_MERGED.pt")
# .pt = weights + live state. fractus/ = the body that ticks.
```

Same-shape checkpoints can be mean-merged. Different `d_model` / layer / expert counts **cannot**. See `docs/CPU_MINI_MERGE_AND_DIMENSIONS.md`.

---

## Train (resume, never from token 0 unless you mean it)

```bash
# offsets: checkpoints/X8_MANIFEST.json or FROZEN_RESUME_MANIFEST.json
CUDA_VISIBLE_DEVICES=0 GPU_ID=0 START_TOKEN=<manifest> \
  BATCH=2 CE_CHUNK=2048 FRACTUS_ATTN_IMPL=chunked BLOCK_CKPT=1 \
  python -u scripts/fast4gpu_boost_v3.py
```

P0 + v4 launcher lives in [fractus-p0](https://huggingface.co/thefinalboss/fractus-p0) (`scripts/fast4gpu_boost_v4.py`).  
PREFIX `unique@40` is the language gate. Do not raise `REPEAT_COEF` above 0.1.

---

## Docs worth reading first

| Doc | Why |
|---|---|
| `docs/EVOLUTION.md` | timestamped history |
| `docs/HOW_FRACTUS_IS_TRAINED.md` | phase-2 recipe |
| `docs/KURAMOTO_BOTTLENECK_AND_FIX.md` | dead experts / 25° arc |
| `docs/OPTIMIZATION_2026-08-22.md` | open-heart kernels |
| `docs/TRAIN_GEN_MISMATCH.md` / p0 `V4_AR.md` | PREFIX vs CARRY |
| `docs/TRUSTED_LOSS.md` | loss ≠ generation |
| `Fractus_White_Paper_v2.md` | architecture write-up |

---

## Citation

```
@misc{robert2026fractus,
  title={Fractus: a Continuous Thought Engine},
  author={Robert, Philippe-Antoine},
  year={2026},
  howpublished={https://huggingface.co/thefinalboss/fractus-cte},
  note={MIT. Dynamical system; linear-attention RNN lineage + phase-routed MoE.}
}
```