Upload docs/LOSS_VS_GEN.md with huggingface_hub

docs/LOSS_VS_GEN.md

@@ -0,0 +1,34 @@
+# Why Train Loss Was Not Accurate vs Generation
+
+**Date:** 2026-08-16 23:31 UTC
+
+## Diagnosis
+
+Teacher-forced CE (train metric) and free-run generation measure different things.
+
+| Metric | GPU1-ish value | Meaning |
+|--------|----------------|---------|
+| Train TF CE (warm continuous) | ~1.3-1.5 | Next-token prediction with ground-truth history + carried thought state |
+| Cold TF CE (reset state) | ~7-40 | Same objective but without warm dynamical state |
+| Free-run AR CE vs ground truth | huge / match~0 | Model leaves data manifold when feeding its own tokens |
+
+So a loss of 1.5 was **accurate for teacher forcing**, not for generation.
+
+## Fix in training
+
+Scheduled sampling (sequential, VRAM-safe):
+
+1. Pass A: teacher-forced CE + LB (always)
+2. Pass B (30% of steps): mix 20% model-sampled tokens into inputs, CE again against true targets
+
+Logged now:
+
+- `tf=` / `ema_tf=` teacher-forced
+- `ss=` / `ema_ss=` scheduled-sampling pass (gen-closer)
+
+Script: `fast4gpu_stage2_ss.py`
+
+## Live (after switch)
+
+GPU1 example: tf~1.45 while ss~2.9 — gap is visible and optimized.
+