File size: 8,627 Bytes
51e7e51 30ccf1f 51e7e51 30ccf1f 51e7e51 fae11c3 51e7e51 fae11c3 51e7e51 fae11c3 51e7e51 30ccf1f 51e7e51 19cb59b b9fd593 51e7e51 d51c2d6 51e7e51 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 | # Fractus-1B Training Log (Live Run)
**Documented:** 2026-08-16 01:37 UTC
## Setup
| Item | Value |
|------|-------|
| Model | Fractus CTE 1B (Continuous Thought Engine) |
| Parameters | 1,048,631,458 |
| Hardware | 4× NVIDIA RTX 5090 |
| Parallelism | 4 independent shard processes (fast4gpu.py) |
| Batch / seq | B=2, seq=128 |
| Precision | bfloat16 autocast |
| Optimizer | SGD (lr=1e-3, momentum=0.9), grad clip 1.0 |
| Architecture | 16 layers, d_model=1280, 20 heads, 128 experts top-k=2, 16 Kuramoto oscillators/block |
| Seed | Progressive growth from fractus_palier0.pt|
## Live metrics (latest)
| GPU | Tokens seen | Loss | Throughput |
|-----|-------------|------|------------|
| 0 | 113,152,000 | **42.4** | 1012 tok/s |
| 1 | 120,396,800 | **33.1** | 1077 tok/s |
| 2 | 121,088,000 | **35.6** | 1083 tok/s |
| 3 | 121,139,200 | **36.7** | 1083 tok/s |
## Loss history (GPU 1 leader, sampled)
| Tokens | Loss |
|--------|------|
| 25,600 | 122.8 |
| 8,601,600 | 118.4 |
| 17,203,200 | 116.8 |
| 25,804,800 | 107.7 |
| 34,406,400 | 91.5 |
| 43,008,000 | 77.4 |
| 51,609,600 | 66.7 |
| 60,211,200 | 58.6 |
| 68,787,200 | 52.4 |
| 77,388,800 | 47.5 |
| 85,990,400 | 43.5 |
| 94,592,000 | 40.3 |
| 103,193,600 | 37.5 |
| 111,795,200 | 35.1 |
| 120,396,800 | 33.1 |
## Per-window descent rate (GPU 1)
| Token window | Approx. rate (loss / M tokens) |
|--------------|--------------------------------|
| 0–30M | 0.76 |
| 30–60M | 1.37 (steepest) |
| 60–90M | 0.56 |
| 90–120M | 0.29 |
| Last ~20M | ~0.27 |
Full-run average rate (GPU1): **~0.75 loss / M tokens** (122.8 → 33.2 over ~120M tokens).
Rate is **still negative** (loss keeps falling) but **decelerating** — expected mid-training behavior, not a stall.
## Generation probe (mid-training, loss ≈ 33)
Checkpoint: `fractus_1b_gpu1.pt` (leader shard).
Loaded **391 / 440** tensors into `ContinuousThoughtEngine` (train-shaped state buffers skipped/reset).
Sampling: temperature 0.9, top-k 50, max 40 new tokens, tick-by-tick CTE decode.
### Raw outputs
```
PROMPT: The meaning of life is
OUTPUT: ibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibibib
PROMPT: Hello, my name is
OUTPUT: iddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddlesiddles
PROMPT: In mathematics, a continuous system
OUTPUT: clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich clich
PROMPT: Fractus thinks
OUTPUT: thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks
```
### Interpretation
- The **generation pipeline works** (embedding → tick loop → logits → sample → decode).
- Outputs show classic **repetition collapse** at this loss level — not coherent language yet.
- For Fractus, loss is **not** assumed to map 1:1 onto GPT-style linguistic quality; coherence will be re-probed near loss 25 and 20.
- Documenting failure modes mid-run is intentional: this is an empirical log, not a marketing card.
## Multimodal note (parallel track)
- tick_vec + fractus/nn/vision.py merged for vision observations.
- CIFAR-10 eyes prototype trained on CPU (fractus_eyes_cifar_final.pt), proving image patches can drive continuous thought.
- Text 1B run was **not** interrupted for eyes work.
## Unified 4-GPU merge generation probe
**Artifact:** `FRACTUS_1B_MERGED_GENTEST.pt` (4.66 GB)
**Method:** mean of live `fractus_1b_gpu{0,1,2,3}.pt` shard checkpoints (strip `_orig_mod.`, reset stateful buffers to batch=1)
**Load:** 424 / 440 tensors into `ContinuousThoughtEngine`
**Date:** 2026-08-16 (mid-run probe)
Same sampling recipe as the single-shard probe (temperature 0.9, top-k 50, max 40 new tokens, tick-by-tick decode).
### Raw outputs (unified merge)
```
PROMPT: The meaning of life is
OUTPUT: uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe
PROMPT: Hello, my name is
OUTPUT: Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado Colorado
PROMPT: In mathematics, a continuous system
OUTPUT: sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails sails
PROMPT: Fractus thinks
OUTPUT: Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population Population
```
### Notes
- Pipeline works on the **averaged** 4-GPU brain, not only on the leader shard.
- Still classic **repetition collapse** at this loss band — not coherent language.
- Token identity differs from GPU1-only probe (`ibib`/`iddles`/`clich`/`thinks` vs `uphe`/`Colorado`/`sails`/`Population`), which is expected: mean weights ≠ best shard.
- Re-probe after further loss drop (targets ~25 and ~20).
## Unified merge probe — loss ≈ 30 (2026-08-16 05:28 UTC)
**Live shard losses at merge time**
| GPU | Tokens | Loss |
|-----|--------|------|
| 0 | 127.9M | 38.6 |
| 1 | 135.7M | **30.0** |
| 2 | 136.4M | 32.3 |
| 3 | 136.0M | 33.5 |
**Artifact:** `FRACTUS_1B_MERGED_LIVE.pt` (4.66 GB) — mean of 4 live shard checkpoints
**Load:** 424 / 440 tensors
### Raw generation outputs (unified)
```
PROMPT: The meaning of life is
OUTPUT: uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe ...
PROMPT: Hello, my name is
OUTPUT: Colorado Colorado Colorado Colorado Colorado Colorado ...
PROMPT: In mathematics, a continuous system
OUTPUT: Fate Fate Fate Fate Fate Fate Fate Fate Fate Fate ...
PROMPT: Fractus thinks
OUTPUT: Population Population Population Population Population ...
PROMPT: Once upon a time
OUTPUT: commands commands commands commands commands commands ...
```
### Reading
- GPU1 crossed **loss 30**.
- Unified checkpoint still produces **word-level tokens** in repetition loops (not char noise).
- New lexical items vs earlier probe (e.g. `Fate`, `commands`) alongside recurring ones (`Colorado`, `Population`).
- Not coherent sentences yet. Pipeline + merge path confirmed again.
- Next probes planned near loss ~25 and ~20.
## Operational policy
- Do not stop the 4-GPU digestion while loss is still monotonically improving.
- Re-merge / re-probe generation at later loss thresholds.
- Throughput stable ~1000–1080 tok/s after 30h+ → no evidence of memory leak in this run.
---
## 2026-08-22/23 — Optimization session (training PAUSED)
No token digestion in this window: code-surgery session instead, on
**AFKmoney/fractus-opt**.
- Phase-2 status: started 2026-08-17T23:26 UTC; paused since ~2026-08-20
with <1% of the pass consumed (~0.7M tok/GPU at last sync).
**Remaining ≈ 425–430M tok/GPU → one full pass ≈ 4.5–5.5 days at the
measured baseline 900–1100 tok/s/GPU** (see HOW_FRACTUS_IS_TRAINED.md §8
for the math and post-opt scenarios).
- Kernels: attention cumsum + chunked proven equivalent to the einsum
reference (fwd/grad/carry); chunked measured ×15–25 at 1B shapes,
memory-flat where the reference crashes.
- Head: memory-flat chunked CE (loss identical within fp32 rounding).
- Data: zero-copy int32 fetch (~27 GB RAM/pod saved).
- Tests: 44/44 (28 repo + 14 equivalence proofs + 2 v2-loop smoke).
---
*Fractus CTE — continuous thought, phase-routed experts, progressive growth.*
*Training log only. Not a claim of finished language competence.* |