Upload docs/TRAINING_LOG_1B.md with huggingface_hub
Browse files- docs/TRAINING_LOG_1B.md +44 -0
docs/TRAINING_LOG_1B.md
CHANGED
|
@@ -130,6 +130,50 @@ OUTPUT: Population Population Population Population Population Population Popul
|
|
| 130 |
- Re-probe after further loss drop (targets ~25 and ~20).
|
| 131 |
|
| 132 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
## Operational policy
|
| 134 |
|
| 135 |
- Do not stop the 4-GPU digestion while loss is still monotonically improving.
|
|
|
|
| 130 |
- Re-probe after further loss drop (targets ~25 and ~20).
|
| 131 |
|
| 132 |
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
## Unified merge probe — loss ≈ 30 (2026-08-16 05:28 UTC)
|
| 136 |
+
|
| 137 |
+
**Live shard losses at merge time**
|
| 138 |
+
|
| 139 |
+
| GPU | Tokens | Loss |
|
| 140 |
+
|-----|--------|------|
|
| 141 |
+
| 0 | 127.9M | 38.6 |
|
| 142 |
+
| 1 | 135.7M | **30.0** |
|
| 143 |
+
| 2 | 136.4M | 32.3 |
|
| 144 |
+
| 3 | 136.0M | 33.5 |
|
| 145 |
+
|
| 146 |
+
**Artifact:** `FRACTUS_1B_MERGED_LIVE.pt` (4.66 GB) — mean of 4 live shard checkpoints
|
| 147 |
+
**Load:** 424 / 440 tensors
|
| 148 |
+
|
| 149 |
+
### Raw generation outputs (unified)
|
| 150 |
+
|
| 151 |
+
```
|
| 152 |
+
PROMPT: The meaning of life is
|
| 153 |
+
OUTPUT: uphe uphe uphe uphe uphe uphe uphe uphe uphe uphe ...
|
| 154 |
+
|
| 155 |
+
PROMPT: Hello, my name is
|
| 156 |
+
OUTPUT: Colorado Colorado Colorado Colorado Colorado Colorado ...
|
| 157 |
+
|
| 158 |
+
PROMPT: In mathematics, a continuous system
|
| 159 |
+
OUTPUT: Fate Fate Fate Fate Fate Fate Fate Fate Fate Fate ...
|
| 160 |
+
|
| 161 |
+
PROMPT: Fractus thinks
|
| 162 |
+
OUTPUT: Population Population Population Population Population ...
|
| 163 |
+
|
| 164 |
+
PROMPT: Once upon a time
|
| 165 |
+
OUTPUT: commands commands commands commands commands commands ...
|
| 166 |
+
```
|
| 167 |
+
|
| 168 |
+
### Reading
|
| 169 |
+
|
| 170 |
+
- GPU1 crossed **loss 30**.
|
| 171 |
+
- Unified checkpoint still produces **word-level tokens** in repetition loops (not char noise).
|
| 172 |
+
- New lexical items vs earlier probe (e.g. `Fate`, `commands`) alongside recurring ones (`Colorado`, `Population`).
|
| 173 |
+
- Not coherent sentences yet. Pipeline + merge path confirmed again.
|
| 174 |
+
- Next probes planned near loss ~25 and ~20.
|
| 175 |
+
|
| 176 |
+
|
| 177 |
## Operational policy
|
| 178 |
|
| 179 |
- Do not stop the 4-GPU digestion while loss is still monotonically improving.
|