Polished professional card: model-details, usage, evaluation-protocol, key-findings, roadmap, limitations

README.md

@@ -16,14 +16,21 @@ datasets:
 
 # GoLLeM-v5 — Tiny English Language Models (16M-32M)
 
-A family of **sub-100M-parameter English language models** trained for the
+Research checkpoints of **sub-100M-parameter English language models**, GPT-style decoders
+(nanoGPT lineage) trained for the
 [Glint Tiny-ML Leaderboard](https://huggingface.co/spaces/Glint-Research/Tiny-ML-Leaderboard).
-GPT-style decoder (nanoGPT lineage), BPE-12k tokenizer, 1024-token context.
-This repository holds training checkpoints from a controlled single-factor token-scaling study.
+This repository is a controlled single-factor scaling study: identical architecture/hyper-parameters/seed,
+varying only tokens and model width.
 
-## Models
+## Model details
 
-All share: BPE-12k tokenizer (`tokenizer.json`, vocab 12288), architecture per size, AdamW (lr 6e-4 -> 6e-5 cosine), seed 1337, bf16.
+- **Architecture:** decoder-only Transformer (nanoGPT lineage), learned positional embeddings, tied input/output embeddings.
+- **Sizes:** 16M variant = 6 layers / d_model 408 / 6 heads (17.4M params); 32M variant = 6 layers / d_model 576 / 9 heads (31.4M params).
+- **Context length:** 1024 tokens.
+- **Tokenizer:** BPE, vocab 12288 (`tokenizer.json`), shared across all checkpoints.
+- **Training:** AdamW, lr 6e-4 -> 6e-5 (cosine), batch 64 x 1024 (65,536 tok/step), seed 1337, bf16 (RTX 5090).
+
+## Checkpoints
 
 | checkpoint | params | shape | tokens | BLiMP | ARC-Easy | WikiText-2 BPB |
 |---|---|---|---|---|---|---|
@@ -32,56 +39,61 @@ All share: BPE-12k tokenizer (`tokenizer.json`, vocab 12288), architecture per s
 | `bpe16m_10B/ckpt.pt`  | 17.4M | L6 d408 h6 | 10B  | 70.36 | 39.52 | 1.1815 |
 | `bpe32m_baseline/ckpt.pt` | 31.4M | L6 d576 h9 | 10B | 70.08 | 42.59 | 1.124 |
 
-**All metrics are computed with the Glint benchmark protocol** (`Glint-1.3/benchmark.py`):
-BLiMP = 67 configs (train split), first-256-token clip, raw sentence log-prob preference;
-ARC-Easy = test split, zero-shot, raw accuracy (`LL(q+choice) - LL(q)`);
-WikiText-2 = byte-normalized bits-per-byte (board's `wiki` field is byte-scale, not tokenizer-token-PPL).
-Note: a generic `lm-eval-harness` run scores BLiMP/ARC ~2-3pp higher than the Glint protocol; the numbers above are the **board-comparable** ones.
+## Usage
 
-## Key findings (single-factor study)
+These are raw nanoGPT-lineage checkpoints (plain `torch` state dicts), **not** `transformers` `AutoModel`
+weights. Architecture is inferable from tensor shapes and the table above.
 
-- **Tokens drive BLiMP, not size.** On the Glint protocol BLiMP keeps climbing with tokens (+1.8pp per doubling, 3.2B->10B) and does **not** plateau; going 16M->32M at matched 10B tokens left BLiMP flat (70.36 -> 70.08).
-- **Size + knowledge drive ARC.** 16M->32M at matched tokens lifted ARC-Easy +3.07pp (39.52 -> 42.59).
-- **Efficiency is size-bonus-weighted**, so the smallest model that reaches a given raw score ranks highest; ARC is the binding lever toward the top of the board (targeted via knowledge distillation, in progress).
+```python
+import torch
+from tokenizers import Tokenizer
+tok   = Tokenizer.from_file("tokenizer.json")          # BPE-12k, vocab 12288
+ckpt  = torch.load("bpe16m_10B/ckpt.pt", map_location="cpu")
+state = ckpt.get("model", ckpt)                        # nanoGPT-style GPT state dict
+# Rebuild a GPT of the tabled shape (e.g. L6 d408 h6, block 1024, tied embeddings),
+# load_state_dict(state), trim logits to vocab 12288, run token ids [B, T].
+```
 
 ## Training data
 
 [`SlayerLab/minimal-en-corpus-5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b)
-— ~5.40B BPE-12k tokens, English, decontaminated. Broad high-quality mix:
+— ~5.40B BPE-12k tokens, English, decontaminated against the benchmark test sets. A broad high-quality mix:
 FineWeb-Edu, DCLM, StackExchange, open-web-math, FineMath, scientific papers, books/Gutenberg, code, CC-News.
-A decontaminated expansion to ~8.3B tokens (added FineWeb-Edu + OpenStax science) is used for later runs.
+A decontaminated expansion to ~8.3B tokens (added FineWeb-Edu + OpenStax science) feeds later runs.
 
-## Usage
+## Evaluation
 
-These are raw nanoGPT-lineage checkpoints (plain `torch` state dicts), **not** `transformers`
-`AutoModel` weights. Architecture is inferable from tensor shapes (embedding `[vocab, d_model]`,
-layer count from keys) and the table above; the tokenizer is `tokenizers`-format.
+All metrics use the **Glint benchmark protocol** (`Glint-1.3/benchmark.py`), i.e. the board-comparable definitions:
 
-```python
-import torch
-from tokenizers import Tokenizer
-tok  = Tokenizer.from_file("tokenizer.json")           # BPE-12k, vocab 12288
-ckpt = torch.load("bpe16m_10B/ckpt.pt", map_location="cpu")
-state = ckpt.get("model", ckpt)                         # nanoGPT-style GPT
-# Rebuild a GPT with the shape from the table (e.g. L6 d408 h6, block 1024),
-# load_state_dict(state), then run token ids [B, T] through the forward.
-```
+- **BLiMP** — 67 configs (train split), each sentence clipped to the first 256 tokens, raw sentence log-prob preference (`good > bad`), no length normalization.
+- **ARC-Easy** — test split, zero-shot, raw accuracy over `LL(question + choice) - LL(question)`.
+- **WikiText-2** — byte-normalized bits-per-byte (the board's `wiki` field is byte-scale; token-perplexity is tokenizer-dependent and not directly comparable across models).
+
+A generic `lm-eval-harness` run scores BLiMP/ARC roughly 2-3pp higher than this protocol; the numbers here are the board-comparable ones.
+
+**Positioning (honest):** leaderboard ranks are **reconstruction estimates** — we reverse-engineered the board
+scoring formula and validated it (it reproduces a published reference model's public rank exactly), then applied it
+to our Glint-protocol metrics. Under that reconstruction the 16M@10B checkpoint sits around #18/74 and the 32M
+baseline around #20/74. These are credible estimates, **not** confirmed entries; an official submission is required to confirm.
+
+## Key findings (single-factor study)
 
-For the exact board-scoring forward (Glint protocol: 256-token clip, raw log-prob), see
-`glint_parity_eval.py` in the project staging.
+- **Tokens drive BLiMP, not size.** BLiMP keeps climbing with tokens (~+1.8pp per doubling, 3.2B->10B) without plateauing on the board protocol; 16M->32M at matched 10B tokens left BLiMP flat.
+- **Capacity + knowledge drive ARC.** 16M->32M at matched tokens lifted ARC-Easy +3.07pp.
+- **Efficiency is size-bonus-weighted**, so the smallest model reaching a given raw score ranks highest; ARC is the binding lever toward the top, targeted next via knowledge distillation.
 
-## Positioning (honest)
+## Roadmap
 
-Leaderboard positions are **reconstruction estimates**: we reverse-engineered and validated the board scoring
-formula (it reproduces a published reference model's rank exactly) and applied it to our Glint-protocol metrics.
-They are credible estimates, **not** confirmed board entries; an official submission is required to confirm.
+- Crown run: 16M @ expanded ~8.3B-token corpus (BLiMP lift from more/better data).
+- ARC boost: knowledge distillation from a strong in-house teacher (decontaminated), plus science-dense data.
+- Larger raw-score variant under evaluation.
 
-## Intended use & limitations
+## Limitations
 
-Research artifacts for small-LM scaling studies and leaderboard work. English-only, base (not instruction-tuned)
-models at 16-32M parameters: expect limited factual knowledge and coherence. Not for production use.
+Base (not instruction-tuned) research models at 16-32M parameters, English-only. Expect limited factual knowledge and
+coherence; not intended for production use.
 
 ## Provenance
 
-Full dialectical record, evaluation artifacts and methodology: labvault `21_09_GoLLeM-v5-Skalowanie-Glint/`
-(including `90-Ewaluacja/EvalHarnessParity.md` for the eval-protocol details). Trained on RunPod RTX 5090.
+Full dialectical record, evaluation artifacts and eval-protocol details in labvault
+`21_09_GoLLeM-v5-Skalowanie-Glint/` (see `90-Ewaluacja/EvalHarnessParity.md`). Trained on RunPod RTX 5090.