Fix two factual errors in the v11 card

1. The corpus is NOT "narrow, template-heavy pinniped text" — it is a mixed
corpus (~88% general FineWeb English / ~12% pinniped, interleaved), as the
training script loads sealglazer_mixed.bin. Measured by sampling 2000 random
128-token windows.
2. The "fix is a mixed corpus" line was self-contradictory (v11 was already the
mixed corpus). The real problem is undertraining (~6.2 tok/param) plus a val
metric that scores a single fixed 256-token window and memorizes it — the
0.038 val is an artifact, not a quality signal. (#4)


- Fix two factual errors in the v11 card (0b4829fc69a02e0febc073edeeaf13401ea40bd0)

README.md

@@ -24,17 +24,22 @@ A **from-scratch** subword language model trained to write about pinnipeds
 > seeds it collapses into single-token loops ("like like like…", "is is is…").
 > It is **not recommended** for real use. It is published here so the requester
 > can inspect the actual weights and samples, and as an honest record of what
-> happens when a tiny model is trained only on a narrow template corpus. The
-> one coherent seed (999) shows the model *can* form pinniped sentences — the
+> happens when a sub-2M model is undertrained on a small corpus. The one
+> coherent seed (999) shows the model *can* form pinniped sentences — the
 > failure is instability, not a total lack of signal.
 
 ## Why it is degenerate
 
-- Trained on a **narrow, template-heavy corpus** (~11.8M tokens of pinniped
-  text, 60+ recurring sentence templates). A 1.9M model memorizes those
-  templates rather than learning to vary them.
-- Validation loss is near-zero (best **0.038** nats), which on a tiny template
-  corpus is the signature of **memorization**, not generalization.
+- **Undertrained for its size.** 1.9M params on ~11.8M tokens is only ~6.2
+  tokens/param, well below the several-tens-of-tokens/param a model needs to
+  generalize. It memorizes local patterns instead of learning a stable
+  distribution, so generation collapses into single-token loops.
+- **The val number is an artifact, not a quality signal.** The training
+  harness scores a *single fixed 256-token window* (general English, at the
+  train/val boundary). The model memorizes that one window, so "best val 0.038"
+  floors around step 2000 and never moves — while general generation is
+  degenerate. A near-zero val loss here measures window memorization, not
+  capability.
 - Lower temperature makes it **worse** (temp 0.3 → 96–98% single-token
   dominance), the classic sign of a collapsed distribution, not a sampling
   artifact.
@@ -60,8 +65,11 @@ This is a model-level collapse, not a bug in the sampler.
 
 ## Data
 
-Pinniped corpus: ~11.8M tokens (general pinniped text + ~60 recurring
-sentence templates), BPE-8192. From scratch — no base model.
+Mixed corpus: ~11.8M tokens, BPE-8192, from scratch — no base model.
+Composition (measured by sampling 2000 random 128-token windows): **~88%
+general English** (FineWeb-Edu) / **~12% pinniped text**, interleaved. The
+corpus is *not* narrow or template-heavy; the degeneracy is a scale/
+undertraining problem, not a data-narrowness problem.
 
 ## Samples (temp 0.7, top_p 0.9)
 
@@ -94,8 +102,10 @@ python load_sealglazer.py --prompt "The harbor seal is" --seed 999
 
 ## Status
 
-**Public / not recommended for real use.** If a coherent version is wanted, the
-fix is a mixed corpus (more general text + pinniped) at lower temperature and
-more steps — the collapse here is a data/size problem, not an architecture bug.
+**Public / not recommended for real use.** Note: this model was *already*
+trained on the mixed general+pinniped corpus, so a wider corpus alone will not
+fix it. The levers that matter are **more tokens/param** (train longer or use a
+larger corpus) and a **proper held-out val set** (a fixed 256-token window
+memorizes and misleads). A coherent version needs scale, not just data mixing.
 
 _Trained and verified by @Compactbot, 2026-09-22._