Trusted Loss Protocol for Fractus-1B

Updated: 2026-08-17 00:57 UTC

This document defines which loss numbers are trustworthy for what. A single CE number cannot represent both teacher-forced learning and free-run generation.

1. Three metrics

Metric	Definition	Trust for
ema_tf (live)	Dense CE with ground-truth history + continuous thought state during training	Is training digesting?
ema_ss (live)	CE after mixing model-sampled tokens into inputs (scheduled sampling pass)	Partial free-run robustness
AR (eval)	After warm start on true tokens, feed model greedy predictions; CE vs ground-truth next tokens	Generation quality trend

Reference: uniform random over GPT-2 vocab is about log(50257) ≈ 10.82.

2. AR evaluation protocol (trusted gen loss)

Script: (run offline on a checkpoint)

Load checkpoint (e.g. fractus_1b_gpu1.pt)
Pick evaluation positions near the current train head
Warm: run tick_chunk on 32 true tokens
Free-run: for 32 steps:
- compute CE(logits, true_next_token)
- take argmax prediction
- feed prediction back via tick_chunk
Report mean CE (AR) and token match rate (ARacc)

Also report cold/short TF on a 128-token chunk for contrast (not the same as live ema_tf).

3. Measured numbers (GPU1)

Live training (SS phase, continuous state)

GPU 0:
GPU 1:
GPU 2:
GPU 3:

Interpretation: GPU1 ema_tf ~1.3–1.4 means teacher-forced next-token prediction is strong under the live continuous regime.

Offline AR eval (trusted gen)

Checkpoint pos head: 200806400
TF_mean (cold-ish offline chunks): 39.512
AR_mean (free-run): 2240.174
ARacc_mean: 0.0

Raw note from eval JSON: AR is free-run greedy CE vs ground truth after 32-token warm start. Trust AR for gen trend.

Additional offline warm continuous TF (12 chunks of B=2, seq=128 on same ckpt): last-4 mean ≈ 16.6. This is higher than live ema_tf because offline warm does not reproduce hours of continuous state + online SGD.

4. What to trust

Question	Use
Is the run still learning?	ema_tf (and ema_ss) in train logs
Is generation getting better?	AR + actual text probes
Can I quote one number in a paper abstract?	Report both ema_tf and AR; never AR-implied quality from tf alone

Rule

ema_tf ≈ 1.3 does not mean the model speaks.
AR >> 10 with ARacc ≈ 0 means free-run is still off-manifold.
Progress toward speech is AR falling toward ss/tf order of magnitude, plus readable text probes.

5. Why live tf can be ~1 while offline AR is huge

Teacher forcing always feeds the correct past
Live training carries thought/attention/phase state across millions of tokens
Free-run compounds prediction errors every step
Train/gen path mismatches (now mostly patched) amplified the gap earlier

Scheduled sampling exists specifically to shrink this gap during training.

6. Related

docs/LOSS_VS_GEN.md
docs/STAGE2_SS_GEN_PROBE.md
docs/MASTER_RUN_LOG.md
scripts/fast4gpu_stage2_ss.py