| # Trusted Loss Protocol for Fractus-1B |
|
|
| **Updated:** 2026-08-17 00:57 UTC |
|
|
| This document defines which loss numbers are trustworthy for what. |
| A single CE number cannot represent both teacher-forced learning and free-run generation. |
|
|
| --- |
|
|
| ## 1. Three metrics |
|
|
| | Metric | Definition | Trust for | |
| |--------|------------|-----------| |
| | **ema_tf** (live) | Dense CE with ground-truth history + continuous thought state during training | Is training digesting? | |
| | **ema_ss** (live) | CE after mixing model-sampled tokens into inputs (scheduled sampling pass) | Partial free-run robustness | |
| | **AR** (eval) | After warm start on true tokens, feed model greedy predictions; CE vs ground-truth next tokens | Generation quality trend | |
|
|
| Reference: uniform random over GPT-2 vocab is about log(50257) ≈ **10.82**. |
|
|
| --- |
|
|
| ## 2. AR evaluation protocol (trusted gen loss) |
|
|
| Script: (run offline on a checkpoint) |
|
|
| 1. Load checkpoint (e.g. fractus_1b_gpu1.pt) |
| 2. Pick evaluation positions near the current train head |
| 3. **Warm:** run tick_chunk on 32 true tokens |
| 4. **Free-run:** for 32 steps: |
| - compute CE(logits, true_next_token) |
| - take argmax prediction |
| - feed prediction back via tick_chunk |
| 5. Report mean CE (AR) and token match rate (ARacc) |
|
|
| Also report cold/short TF on a 128-token chunk for contrast (not the same as live ema_tf). |
| |
| --- |
| |
| ## 3. Measured numbers (GPU1) |
| |
| ### Live training (SS phase, continuous state) |
| |
| - GPU 0: |
| - GPU 1: |
| - GPU 2: |
| - GPU 3: |
| |
| Interpretation: GPU1 **ema_tf ~1.3–1.4** means teacher-forced next-token prediction is strong under the live continuous regime. |
| |
| ### Offline AR eval (trusted gen) |
| |
| - Checkpoint pos head: 200806400 |
| - TF_mean (cold-ish offline chunks): **39.512** |
| - **AR_mean (free-run): 2240.174** |
| - **ARacc_mean: 0.0** |
|
|
| Raw note from eval JSON: AR is free-run greedy CE vs ground truth after 32-token warm start. Trust AR for gen trend. |
|
|
| Additional offline warm continuous TF (12 chunks of B=2, seq=128 on same ckpt): last-4 mean ≈ **16.6**. |
| This is higher than live ema_tf because offline warm does not reproduce hours of continuous state + online SGD. |
| |
| --- |
| |
| ## 4. What to trust |
| |
| | Question | Use | |
| |----------|-----| |
| | Is the run still learning? | **ema_tf** (and ema_ss) in train logs | |
| | Is generation getting better? | **AR** + actual text probes | |
| | Can I quote one number in a paper abstract? | Report **both** ema_tf and AR; never AR-implied quality from tf alone | |
| |
| ### Rule |
| |
| - **ema_tf ≈ 1.3** does **not** mean the model speaks. |
| - **AR >> 10 with ARacc ≈ 0** means free-run is still off-manifold. |
| - Progress toward speech is **AR falling** toward ss/tf order of magnitude, plus readable text probes. |
| |
| --- |
| |
| ## 5. Why live tf can be ~1 while offline AR is huge |
| |
| 1. Teacher forcing always feeds the correct past |
| 2. Live training carries thought/attention/phase state across millions of tokens |
| 3. Free-run compounds prediction errors every step |
| 4. Train/gen path mismatches (now mostly patched) amplified the gap earlier |
| |
| Scheduled sampling exists specifically to shrink this gap during training. |
| |
| --- |
| |
| ## 6. Related |
| |
| - docs/LOSS_VS_GEN.md |
| - docs/STAGE2_SS_GEN_PROBE.md |
| - docs/MASTER_RUN_LOG.md |
| - scripts/fast4gpu_stage2_ss.py |
|
|
