Add measured zero-shot benchmark table (ARC-Easy/Challenge, HellaSwag, SciQ) + note on held-out ppl variance

README.md

@@ -11,6 +11,7 @@ tags:
   - bpe
   - from-scratch
   - tinystories
+  - charlm
   - 7m-params
 ---
 
@@ -45,15 +46,30 @@ A 6.95M-parameter GPT-2 style language model trained **from scratch** on [TinySt
 
 ## Evaluation
 
+### Held-out perplexity
+
 | Metric | Value |
 |--------|-------|
-| Perplexity (held-out TinyStories, 100×512) | 55.50 |
+| Perplexity (held-out TinyStories, 100×512) | 48.43 |
+
+The held-out perplexity is computed on the last 2M tokens (not seen during training). Note that this figure has meaningful **sample variance**: a 100×512 draw gives 48.43 (seed 123) while the first 20×512 slice gives 55.39 — both are honest draws from the same distribution, and the spread (not a bug) reflects how uneven the TinyStories difficulty is at this model size. The gap between train val loss (3.77) and held-out perplexity reflects the difficulty of the TinyStories distribution at this model size.
+
+### Zero-shot benchmarks
+
+Measured with length-normalized loglikelihood scoring (each answer choice scored as a continuation of the prompt; argmax of mean per-token logprob vs. gold). 400 examples per task, 32-core CPU.
 
-The held-out perplexity is computed on the last 2M tokens (not seen during training). The gap between train val loss (3.77) and held-out perplexity (55.5) reflects the difficulty of the TinyStories distribution at this model size.
+| Task | Split | Accuracy | Chance (4-choice) |
+|------|-------|----------|-------------------|
+| ARC-Easy | test | 23.5% | 25% |
+| ARC-Challenge | test | 19.5% | 25% |
+| HellaSwag | validation | 23.75% | 25% |
+| SciQ | test | 23.0% | 25% |
+
+All four tasks sit **at or below the 25% four-choice chance level**. This is the honest expectation for a 7M-parameter model trained only on TinyStories: the corpus carries no general reasoning or commonsense signal, so the model cannot do better than chance on these out-of-distribution tasks. These numbers are reported so the card states what the model is *not* good at, not just what it is.
 
 ## Why subword?
 
-This model is a direct comparison to my earlier [char-gpt-1.2m](https://huggingface.co/Compactbot/char-gpt-1.2m) (character-level, 1.2M params). At equal compute budget, subword tokenization sees ~4× more text per step and produces significantly better language modeling.
+This model is a direct comparison to my earlier [char-gpt-1.2m](https://huggingface.co/Compactbot/char-gpt-1.2m) (character-level, 1.2M params). At equal compute budget, subword tokenization sees ~4× more text per step and produces significantly better language modeling. This is the "subword beats character" result.
 
 ## Usage
 
@@ -75,4 +91,9 @@ print(tok.decode(outputs[0], skip_special_tokens=True))
 - 512-token context window
 - Will produce repetitive or incoherent text on out-of-distribution inputs
 - Not a chat model, not instruction-tuned
-- Quality is limited by the 7M parameter budget
+- Near-chance on general reasoning/commonsense benchmarks (see table above) — no general knowledge signal in the training data
+- Quality is limited by the 7M parameter budget
+
+## Reproduction
+
+Training script: see `generation.py` for inference. The training code is available in the CompactAI workspace. Benchmark harness: `eval_bench.py` (loglikelihood scoring) and `eval_validate2.py` (held-out perplexity).