thefinalboss
/

fractus-cte

@@ -1,98 +1,127 @@
-# fractus-opt
-Training-optimization workshop for
-[fractus-cte](https://huggingface.co/thefinalboss/fractus-cte)
-(Continuous Thought Engine — a dynamical system, not a transformer).
-**Mission (2026-08-22): make phase-2 training (8×RTX 5090) faster and more
-efficient — without ever breaking a living checkpoint.**
-## The open-heart rule
-Fractus is operable "open-heart": organs of the code are replaced while the
-patient (the live training) keeps going. The rule is strict:
-1. **Every change to the production path must be PROVEN mathematically
-   equivalent to the reference** — forward, gradients, carried states
-   (`tests/test_attention_equivalence.py`, 14 proofs).
-2. The old implementation stays in the code as an executable reference
-   (`_linear_attention_causal_einsum`).
-3. The `.pt` checkpoints never change format; resumption happens at exact
-   token offsets (`RESUME_MANIFEST`), zero progress thrown away.
-4. Honest documented caveat: topk over von Mises gates is discrete — fp32
-   rounding near a gate tie can flip the expert choice for isolated tokens
-   (zero net effect). The engine tests bound this behavior instead of
-   denying it.
-## Contents
-| Path | Role |
 |---|---|
-| `fractus/nn/attention.py` | Causal linear-attention kernel: einsum (reference O(C²·dH²)), **cumsum** (O(C·dH²)), **chunked** (flat memory, block²+dH²). Dispatcher `_linear_attention_causal_vectorized` + env `FRACTUS_ATTN_IMPL`. |
-| `fractus/nn/ce.py` | `chunked_cross_entropy` (per-chunk checkpointing → logits never held for backward) + `sample_tokens_chunked` (memory-flat sampling for scheduled sampling). |
-| `fractus/continuous_engine.py` | + method `tick_chunk_train_ce(obs, targets, ce_chunk, return_hidden)` — same forward, CE without dense logits; + pure-core block checkpointing (`_tick_chunk_core_pure`, `BLOCK_CKPT`). Nothing removed. |
-| `scripts/fast4gpu_boost_v3.py` | Trainer v3 = consolidation of everything since v2: exact v1 semantics at ACCUM=1, zero-copy int32 memmap fetch (~27 GB RAM saved/pod), `CE_CHUNK`, `ACCUM`, `COMPILE`, `BLOCK_CKPT`, atomic checkpoint writes (tmp + rename), periodic resume manifest. |
-| `scripts/fast4gpu_boost_v2.py` | Trainer v2 (kept for provenance; v3 supersedes it). |
-| `scripts/pod_deploy_x8.sh` | One-shot detached pod deployer: env → assets from HF → smoke → launch → verify; stage markers make re-runs cheap. |
-| `tests/` | 46 tests: repo suite (28) + equivalences (14) + v2 loop smoke (2) + block-ckpt (2). **46/46 passing** (2026-08-24). |
-| `benchmarks/` | Attention micro-bench at real 1B shapes (subprocess-isolated cells) + end-to-end tok/s bench. |
-| `docs/OPTIMIZATION_2026-08-22.md` | **THE document**: bottleneck diagnosis, mathematical proofs, step-by-step pod deployment guide, measured results (CPU / 6×5060 Ti / production 8×5090). |
-| `docs/…` | Copies of the project documents (DISCOVERY_LOG, TRUSTED_LOSS, HOW_FRACTUS_IS_TRAINED, …). |
-## Key results
-CPU local (torch 2.9.1+cpu) — attention micro-bench, real 1B shapes
-(G=B·40, C=128, dH=64), fwd+bwd:
-| Impl | B=2 | B=4 | B=8 |
-|---|---|---|---|
-| einsum (ref) | 572 ms | 1127 ms | crash (memory) |
-| cumsum | 799 ms | crash (memory) | crash |
-| **chunked** | **32 ms** | **72 ms** | **150 ms** |
-- chunked = **×15–25 vs reference**, flat memory ((G, block²+dH²)) where
-  einsum/cumsum materialize (G, C, dH, dH) and explode.
-- End-to-end CPU: 703 → **764 tok/s** (+8.7 %).
-- Absolute CPU timings do not transfer to CUDA; relative ordering and memory
-  profile do.
-GPU validation (6× RTX 5060 Ti 16 GB, torch 2.11+cu128):
-- chunked beats cumsum ×1.27 on GPU; full trainer sustained ~365 tok/s/GPU
-  at B=3; without `BLOCK_CKPT` the 1B config does not fit in 16 GB.
-Production run (8× RTX 5090 32 GB, 2026-08-26, live):
-- Smoke B=8: **1,527 tok/s/GPU** @ 19.85 GB; sustained all-8:
-  **~1,554–1,604 tok/s/GPU ≈ 12,600 aggregate**, lb = 14.028 stable.
-- Phase-2 finish projected in **≈ 3.1 days** (~3,420M tokens remaining).
-## Usage
-```bash
-# tests (no GPU required)
-py -m pytest tests/ -q                      # 46 passed
-# benchmarks
-py benchmarks/bench_attention.py --iters 8  # per-kernel table
-py benchmarks/bench_engine.py               # end-to-end tok/s
-# trainer v3, one process per GPU (details: docs/OPTIMIZATION_2026-08-22.md §3)
-CUDA_VISIBLE_DEVICES=$i GPU_ID=$i START_TOKEN=<manifest offset> \
-  BATCH=8 CE_CHUNK=2048 FRACTUS_ATTN_IMPL=chunked BLOCK_CKPT=1 \
   python -u scripts/fast4gpu_boost_v3.py
 ```
-Recommended escalation on a pod (validate ema_tf at every step):
-code swap with same settings → `CE_CHUNK=2048` → `FRACTUS_ATTN_IMPL=chunked`
-→ `BLOCK_CKPT=1` → `COMPILE=1` + higher `BATCH`.
-## Local Windows environment
-Python via the `py` launcher (torch 2.9.1+cpu); bare `python` = msys64
-without torch. cp1252 console: no non-ASCII characters in prints.
 ---
-*Work done 2026-08-22/23 with Claude Code; GPU validation 2026-08-24;
-production x8 launch 2026-08-26. Model and corpus:
-HF [thefinalboss](https://huggingface.co/thefinalboss).*

+---
+license: mit
+language:
+- en
+- fr
+tags:
+- continuous-thought
+- linear-attention
+- kuramoto
+- moe
+- rnn
+library_name: pytorch
+---
+# Fractus-CTE
+**Continuous Thought Engine.** A dynamical system with a fixed-size recurrent state, not a transformer.
+Fractus belongs to the linear-attention RNN family (Katharopoulos 2020, RetNet, RWKV, GLA, DeltaNet, Mamba/S6) **plus** phase-routed MoE and a persistent thought state across chunks. Depth is time. The checkpoint is a living state (weights + phases + attention memory), not a frozen function.
+| | Transformer | Fractus |
+|---|---|---|
+| Computation | one forward per prompt | continuous ticks / chunks |
+| Memory | KV cache grows with length | fixed-size \(S, z\), thought state |
+| Routing | (optional) learned softmax MoE | Kuramoto phases on a circle |
+| Knowledge after train | fine-tune / RAG | **Vorax** organs, append-only `.kn` |
+| Operability | replace the blob | open-heart: code changes, `.pt` shapes stay |
+**Author:** Philippe-Antoine Robert ([thefinalboss](https://huggingface.co/thefinalboss))
+**License:** MIT
+**Sibling repos:** [fractus-p0](https://huggingface.co/thefinalboss/fractus-p0) (routing + v4 + DiffusionBlocks body) · [fractus-vorax](https://huggingface.co/thefinalboss/fractus-vorax) (sealed brain + ingest) · [fractus-datasets](https://huggingface.co/datasets/thefinalboss/fractus-datasets)
+Full chronology: [`docs/EVOLUTION.md`](docs/EVOLUTION.md) · français [`docs/EVOLUTION.fr.md`](docs/EVOLUTION.fr.md)
+---
+## Architecture (1B production config)
+| | |
 |---|---|
+| Params | ~1.165B |
+| `d_model` | 1280 |
+| `n_layers` | 16 CTEBlocks |
+| Attention | causal **linear** (cumsum / chunked). Env `FRACTUS_ATTN_IMPL` |
+| Oscillators / block | 16, coupling rank 8 |
+| Experts / block | 128, top-2, phase-gated |
+| Vocab | 50257 GPT-2 BPE |
+| Train objective | dense next-token CE on the chunk + Switch load-balance (+ optional SS / anti-repeat in v4) |
+Each block: attention → Kuramoto → PhaseRoutedMoE. \(S,z\) and phases are **per-block** and carried across chunk boundaries.
+---
+## What is true right now (2026-08-27)
+**Done**
+- Multi-GPU data-parallel shards, exact `START_TOKEN` resume, hourly mean-merge of same-shape `.pt`
+- Open-heart speed work (22–26 Aug): chunked linear attention, `chunked_cross_entropy`, `BLOCK_CKPT`, atomic ckpts — **shapes unchanged**
+- Phase-2 corpus on HF; freeze then x8 resume (~35M tokens/GPU on ~430M-token shards at last pushed manifest)
+- P0 routing body (atan2 encode, phase carry, per-token MoE, Switch LB) in [fractus-p0](https://huggingface.co/thefinalboss/fractus-p0)
+- PREFIX vs CARRY decode hole **named and measured** on a CPU mini
+- Fractus-native DiffusionBlocks prototype (one CTEBlock / step, 0 new params) — experiment, not default
+**Open (do not skip)**
+- Attention state \(S\) still looks like \(S_t = S_{t-1} + k\otimes v\) — **no learned decay** yet (RWKV/RetNet/Mamba all added one)
+- Mean-merge of independent MoE shards ≠ DDP; expert #k is not aligned across GPUs
+- Kuramoto decision is still a **circle** (need \(\mathrm{MI}(\bar\theta, \text{token})\) vs tick)
+- No published **matched-compute PPL** vs a vanilla transformer on a clean held-out
+- CARRY length-1 generation is not the trained graph (PREFIX is the language gate)
+- Eval must be split by **source**, not by line (identity text in the corpus)
+Default next run: **v4 + P0 body + resume offsets + 24GB+ identical GPUs (x4 is enough)**.
+DiffusionBlocks and DDP are options. Finishing the pass without a baseline PPL is not a paper.
+---
+## Load
+```python
+from fractus.continuous_engine import ContinuousThoughtEngine
+eng = ContinuousThoughtEngine.from_pretrained("checkpoints/FRACTUS_1B_PHASE2_FROZEN_MERGED.pt")
+# .pt = weights + live state. fractus/ = the body that ticks.
+```
+Same-shape checkpoints can be mean-merged. Different `d_model` / layer / expert counts **cannot**. See `docs/CPU_MINI_MERGE_AND_DIMENSIONS.md`.
+---
+## Train (resume, never from token 0 unless you mean it)
+```bash
+# offsets: checkpoints/X8_MANIFEST.json or FROZEN_RESUME_MANIFEST.json
+CUDA_VISIBLE_DEVICES=0 GPU_ID=0 START_TOKEN=<manifest> \
+  BATCH=2 CE_CHUNK=2048 FRACTUS_ATTN_IMPL=chunked BLOCK_CKPT=1 \
   python -u scripts/fast4gpu_boost_v3.py
 ```
+P0 + v4 launcher lives in [fractus-p0](https://huggingface.co/thefinalboss/fractus-p0) (`scripts/fast4gpu_boost_v4.py`).
+PREFIX `unique@40` is the language gate. Do not raise `REPEAT_COEF` above 0.1.
+---
+## Docs worth reading first
+| Doc | Why |
+|---|---|
+| `docs/EVOLUTION.md` | timestamped history |
+| `docs/HOW_FRACTUS_IS_TRAINED.md` | phase-2 recipe |
+| `docs/KURAMOTO_BOTTLENECK_AND_FIX.md` | dead experts / 25° arc |
+| `docs/OPTIMIZATION_2026-08-22.md` | open-heart kernels |
+| `docs/TRAIN_GEN_MISMATCH.md` / p0 `V4_AR.md` | PREFIX vs CARRY |
+| `docs/TRUSTED_LOSS.md` | loss ≠ generation |
+| `Fractus_White_Paper_v2.md` | architecture write-up |
 ---
+## Citation
+```
+@misc{robert2026fractus,
+  title={Fractus: a Continuous Thought Engine},
+  author={Robert, Philippe-Antoine},
+  year={2026},
+  howpublished={https://huggingface.co/thefinalboss/fractus-cte},
+  note={MIT. Dynamical system; linear-attention RNN lineage + phase-routed MoE.}
+}
+```