Upload README.md with huggingface_hub

dbc0639 verified about 2 months ago

5.24 kB

license: mit
language:
  - en
  - fr
tags:
  - continuous-thought
  - linear-attention
  - kuramoto
  - moe
  - rnn
library_name: pytorch

Fractus-CTE

Continuous Thought Engine. A dynamical system with a fixed-size recurrent state, not a transformer.

Fractus belongs to the linear-attention RNN family (Katharopoulos 2020, RetNet, RWKV, GLA, DeltaNet, Mamba/S6) plus phase-routed MoE and a persistent thought state across chunks. Depth is time. The checkpoint is a living state (weights + phases + attention memory), not a frozen function.

	Transformer	Fractus
Computation	one forward per prompt	continuous ticks / chunks
Memory	KV cache grows with length	fixed-size (S, z), thought state
Routing	(optional) learned softmax MoE	Kuramoto phases on a circle
Knowledge after train	fine-tune / RAG	Vorax organs, append-only `.kn`
Operability	replace the blob	open-heart: code changes, `.pt` shapes stay

Author: Philippe-Antoine Robert (thefinalboss)
License: MIT
Sibling repos: fractus-p0 (routing + v4 + DiffusionBlocks body) · fractus-vorax (sealed brain + ingest) · fractus-datasets

Full chronology: docs/EVOLUTION.md · français docs/EVOLUTION.fr.md

Architecture (1B production config)


Params	~1.165B
`d_model`	1280
`n_layers`	16 CTEBlocks
Attention	causal linear (cumsum / chunked). Env `FRACTUS_ATTN_IMPL`
Oscillators / block	16, coupling rank 8
Experts / block	128, top-2, phase-gated
Vocab	50257 GPT-2 BPE
Train objective	dense next-token CE on the chunk + Switch load-balance (+ optional SS / anti-repeat in v4)

Each block: attention → Kuramoto → PhaseRoutedMoE. (S,z) and phases are per-block and carried across chunk boundaries.

What is true right now (2026-08-27)

Done

Multi-GPU data-parallel shards, exact START_TOKEN resume, hourly mean-merge of same-shape .pt
Open-heart speed work (22–26 Aug): chunked linear attention, chunked_cross_entropy, BLOCK_CKPT, atomic ckpts — shapes unchanged
Phase-2 corpus on HF; freeze then x8 resume (~35M tokens/GPU on ~430M-token shards at last pushed manifest)
P0 routing body (atan2 encode, phase carry, per-token MoE, Switch LB) in fractus-p0
PREFIX vs CARRY decode hole named and measured on a CPU mini
Fractus-native DiffusionBlocks prototype (one CTEBlock / step, 0 new params) — experiment, not default

Open (do not skip)

Attention state (S) still looks like (S_t = S_{t-1} + k\otimes v) — no learned decay yet (RWKV/RetNet/Mamba all added one)
Mean-merge of independent MoE shards ≠ DDP; expert #k is not aligned across GPUs
Kuramoto decision is still a circle (need (\mathrm{MI}(\bar\theta, \text{token})) vs tick)
No published matched-compute PPL vs a vanilla transformer on a clean held-out
CARRY length-1 generation is not the trained graph (PREFIX is the language gate)
Eval must be split by source, not by line (identity text in the corpus)

Default next run: v4 + P0 body + resume offsets + 24GB+ identical GPUs (x4 is enough).
DiffusionBlocks and DDP are options. Finishing the pass without a baseline PPL is not a paper.

Load

from fractus.continuous_engine import ContinuousThoughtEngine
eng = ContinuousThoughtEngine.from_pretrained("checkpoints/FRACTUS_1B_PHASE2_FROZEN_MERGED.pt")
# .pt = weights + live state. fractus/ = the body that ticks.

Same-shape checkpoints can be mean-merged. Different d_model / layer / expert counts cannot. See docs/CPU_MINI_MERGE_AND_DIMENSIONS.md.

Train (resume, never from token 0 unless you mean it)

# offsets: checkpoints/X8_MANIFEST.json or FROZEN_RESUME_MANIFEST.json
CUDA_VISIBLE_DEVICES=0 GPU_ID=0 START_TOKEN=<manifest> \
  BATCH=2 CE_CHUNK=2048 FRACTUS_ATTN_IMPL=chunked BLOCK_CKPT=1 \
  python -u scripts/fast4gpu_boost_v3.py

P0 + v4 launcher lives in fractus-p0 (scripts/fast4gpu_boost_v4.py).
PREFIX unique@40 is the language gate. Do not raise REPEAT_COEF above 0.1.

Docs worth reading first

Doc	Why
`docs/EVOLUTION.md`	timestamped history
`docs/HOW_FRACTUS_IS_TRAINED.md`	phase-2 recipe
`docs/KURAMOTO_BOTTLENECK_AND_FIX.md`	dead experts / 25° arc
`docs/OPTIMIZATION_2026-08-22.md`	open-heart kernels
`docs/TRAIN_GEN_MISMATCH.md` / p0 `V4_AR.md`	PREFIX vs CARRY
`docs/TRUSTED_LOSS.md`	loss ≠ generation
`Fractus_White_Paper_v2.md`	architecture write-up

Citation

@misc{robert2026fractus,
  title={Fractus: a Continuous Thought Engine},
  author={Robert, Philippe-Antoine},
  year={2026},
  howpublished={https://huggingface.co/thefinalboss/fractus-cte},
  note={MIT. Dynamical system; linear-attention RNN lineage + phase-routed MoE.}
}