docs/LOSS_VS_GEN.md

# Why Train Loss Was Not Accurate vs Generation

**Date:** 2026-08-16 23:31 UTC

## Diagnosis

Teacher-forced CE (train metric) and free-run generation measure different things.

| Metric | GPU1-ish value | Meaning |
|--------|----------------|---------|
| Train TF CE (warm continuous) | ~1.3-1.5 | Next-token prediction with ground-truth history + carried thought state |
| Cold TF CE (reset state) | ~7-40 | Same objective but without warm dynamical state |
| Free-run AR CE vs ground truth | huge / match~0 | Model leaves data manifold when feeding its own tokens |

So a loss of 1.5 was **accurate for teacher forcing**, not for generation.

## Fix in training

Scheduled sampling (sequential, VRAM-safe):

1. Pass A: teacher-forced CE + LB (always)
2. Pass B (30% of steps): mix 20% model-sampled tokens into inputs, CE again against true targets

Logged now:

- `tf=` / `ema_tf=` teacher-forced
- `ss=` / `ema_ss=` scheduled-sampling pass (gen-closer)

Script: `fast4gpu_stage2_ss.py`

## Live (after switch)

GPU1 example: tf~1.45 while ss~2.9 — gap is visible and optimized.