File size: 1,148 Bytes
e24216a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | # Live Gen Probe 8-GPU
Updated: 2026-08-17T19:29:56.573368+00:00
- ckpt:
- loaded tensors: 391
## Prompt
- unique tokens: 5
## Prompt
- unique tokens: 6
## Prompt
- unique tokens: 5
## Prompt
- unique tokens: 4
## Prompt
- unique tokens: 6
## Training snapshot (same time)
| GPU | tokens | ema_tf | tok/s |
|-----|--------|--------|-------|
| 0 | ~8.9M | ~9.1 | ~955 |
| 1 | ~8.7M | ~8.9 | ~933 |
| 2 | ~9.9M | ~8.0 | ~1062 |
| 3 | ~8.7M | ~8.7 | ~932 |
| 4 | ~8.8M | ~9.4 | ~941 |
| 5 | ~9.2M | ~9.2 | ~991 |
| 6 | ~9.0M | ~8.9 | ~960 |
| 7 | ~9.4M | ~8.6 | ~1010 |
## Interpretation
- Teacher-forced loss continues to fall on all 8 GPUs (ema_tf ~8-9 after ~9M tokens/shard).
- Free-run greedy generation still shows **single-token attractor collapse** (repeat last word / commas).
- This matches prior stage findings: knowledge in weights (TF) != free-run coherence yet (AR gap / decode dynamics).
- Unique tokens per prompt ~4-6 out of 48 generated.
## Next
- Keep SS training running.
- Optional: re-apply decode surgery / temperature / n-gram ban at inference.
- Do not read TF loss alone as "model speaks".
|