Proper English model card: Glint-protocol metrics, honest positioning, corrected corpus/positions

README.md

@@ -2,108 +2,67 @@
 license: apache-2.0
 language:
 - en
+library_name: pytorch
+pipeline_tag: text-generation
 tags:
 - tiny-lm
-- babylm
-- language-model
-- from-scratch
 - gpt
-library_name: pytorch
-pipeline_tag: text-generation
+- nanogpt
+- glint-tiny-ml-leaderboard
+- english
+datasets:
+- SlayerLab/minimal-en-corpus-5b
 ---
 
-# GoLLeM-v5 — English Tiny-LM Checkpoints
-
-Training checkpoints for **GoLLeM-v5**, a family of small English language models
-(≈17M–31M parameters) trained **from scratch** on a curated, decontaminated English
-corpus. Built for the **sub-100M efficiency regime** and evaluated on the tiny-ML
-leaderboard tasks: **BLiMP**, **ARC-Easy**, and **WikiText-2**.
-
-Trained on a single RunPod **RTX 5090**. Redundant backups: this HF repo (durable) +
-the training pod (origin) + local disk.
-
-## Architecture
-
-All 16M checkpoints share one decoder-only GPT (`GPT-ref`):
-
-| | |
-|---|---|
-| Layers / width / heads | **6 / 408 / 6** (≈17.4M params) |
-| Context length | 1024 |
-| Vocabulary | 12288 (BPE-12k) |
-| Optimizer | AdamW, lr 6e-4 → 6e-5 (cosine) |
-| Precision / seed | bf16 / 1337 |
-
-The 32M baseline is the same recipe at **L6 / d576 / h9 (≈31.4M params)**.
-
-## Checkpoints
-
-### Token-scaling scan (16M, single-factor: identical arch/hypers, only the token budget changes)
-
-| file | tokens | epochs | BLiMP | ARC-Easy | WikiText-2 (BPB) |
-|------|-------:|-------:|------:|---------:|-----------------:|
-| `bpe16m_3.2B/ckpt.pt` | 3.2B | 1.2 | 71.54 | 39.39 | 1.2161 |
-| `bpe16m_6B/ckpt.pt`   | 6B   | 2.2 | 73.29 | 41.04 | 1.1943 |
-| `bpe16m_10B/ckpt.pt`  | 10B  | 3.7 | 73.43 | 41.67 | 1.1815 |
-
-Metrics above are from `lm-evaluation-harness` (BLiMP full, ARC-Easy, WikiText-2 bits-per-byte).
-
-**Finding:** BLiMP saturates around ~6B tokens for this 16M model (Δ 6→10B = +0.14);
-ARC-Easy keeps improving at 10B; BPB decreases monotonically.
-
-### 32M size-isolation baseline
+# GoLLeM-v5 — Tiny English Language Models (16M-32M)
 
-| file | params | tokens |
-|------|-------:|-------:|
-| `bpe32m_baseline/ckpt.pt` | 31.4M (L6/d576/h9) | 10B |
+A family of **sub-100M-parameter English language models** trained for the
+[Glint Tiny-ML Leaderboard](https://huggingface.co/spaces/Glint-Research/Tiny-ML-Leaderboard).
+GPT-style decoder (nanoGPT lineage), BPE-12k tokenizer, 1024-token context.
+This repository holds training checkpoints from a controlled single-factor token-scaling study.
 
-### Leaderboard protocol (Glint) results
+## Models
 
-The public leaderboard uses a stricter evaluation protocol (256-token clip, no BOS,
-train-split BLiMP, bare-prompt ARC-Easy raw accuracy, byte-normalized WikiText). Under
-that protocol our numbers are lower than the `lm-eval` numbers above (offset ≈ 3.1pp
-BLiMP / 2.2pp ARC) — **always cite the Glint numbers for the board**:
+All share: BPE-12k tokenizer (`tokenizer.json`, vocab 12288), architecture per size, AdamW (lr 6e-4 -> 6e-5 cosine), seed 1337, bf16.
 
-| model | tokens | BLiMP | ARC-Easy | efficiency rank |
-|-------|-------:|------:|---------:|:---------------:|
-| 16M | 10B | 70.36 | 39.52 | #18 / 74 |
-| 32M | 10B | 70.08 | 42.59 | #20 / 74 |
+| checkpoint | params | shape | tokens | BLiMP | ARC-Easy | WikiText-2 BPB |
+|---|---|---|---|---|---|---|
+| `bpe16m_3.2B/ckpt.pt` | 17.4M | L6 d408 h6 | 3.2B | 67.40 | 38.22 | 1.2161 |
+| `bpe16m_6B/ckpt.pt`   | 17.4M | L6 d408 h6 | 6B   | 68.92 | 39.10 | 1.1943 |
+| `bpe16m_10B/ckpt.pt`  | 17.4M | L6 d408 h6 | 10B  | 70.36 | 39.52 | 1.1815 |
+| `bpe32m_baseline/ckpt.pt` | 31.4M | L6 d576 h9 | 10B | 70.08 | 42.59 | 1.124 |
 
-The 32M model ranks *below* the 16M model on efficiency: the leaderboard's size bonus
-favors smaller models, and the extra ARC gain does not offset the reduced bonus — hence
-the crown effort stays at the 16M size.
+**All metrics are computed with the Glint benchmark protocol** (`Glint-1.3/benchmark.py`):
+BLiMP = 67 configs (train split), first-256-token clip, raw sentence log-prob preference;
+ARC-Easy = test split, zero-shot, raw accuracy (`LL(q+choice) - LL(q)`);
+WikiText-2 = byte-normalized bits-per-byte (board's `wiki` field is byte-scale, not tokenizer-token-PPL).
+Note: a generic `lm-eval-harness` run scores BLiMP/ARC ~2-3pp higher than the Glint protocol; the numbers above are the **board-comparable** ones.
 
-## Tokenizer
+## Key findings (single-factor study)
 
-`tokenizer.json` — BPE, vocab 12288 ("BPE-12k"), shared by every v5 model. Locked after a
-3/3 verdict against a byte-level baseline (BLiMP +2σ ∧ WikiText-2 BPB Δ0.238 ∧ improved
-leaderboard efficiency).
+- **Tokens drive BLiMP, not size.** On the Glint protocol BLiMP keeps climbing with tokens (+1.8pp per doubling, 3.2B->10B) and does **not** plateau; going 16M->32M at matched 10B tokens left BLiMP flat (70.36 -> 70.08).
+- **Size + knowledge drive ARC.** 16M->32M at matched tokens lifted ARC-Easy +3.07pp (39.52 -> 42.59).
+- **Efficiency is size-bonus-weighted**, so the smallest model that reaches a given raw score ranks highest; ARC is the binding lever toward the top of the board (targeted via knowledge distillation, in progress).
 
 ## Training data
 
-- Base corpus: **`SlayerLab/minimal-en-corpus-5b`** — 5.40B unique BPE-12k tokens, a
-  15-source English mixture (FineWeb-Edu, DCLM, code, math, books, QA, science),
-  decontaminated against the benchmark test sets.
-- An **expanded 8.29B-token** corpus (base + additional FineWeb-Edu-100BT + OpenStax
-  science, all deduplicated and decontaminated) backs the in-progress 16M crown run.
+[`SlayerLab/minimal-en-corpus-5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-5b)
+— ~5.40B BPE-12k tokens, English, decontaminated. Broad high-quality mix:
+FineWeb-Edu, DCLM, StackExchange, open-web-math, FineMath, scientific papers, books/Gutenberg, code, CC-News.
+A decontaminated expansion to ~8.3B tokens (added FineWeb-Edu + OpenStax science) is used for later runs.
 
-## Usage
+## Positioning (honest)
 
-```python
-import torch
-ckpt = torch.load("bpe16m_10B/ckpt.pt", map_location="cpu")
-sd = ckpt["model"] if "model" in ckpt else ckpt
-# GPT-ref: L6 d408 h6, vocab 12288, block 1024.
-# Model class + forward: train_gpt_ref.py. Trim logits to vocab 12288 (no padded vocab).
-```
+Leaderboard positions are **reconstruction estimates**: we reverse-engineered and validated the board scoring
+formula (it reproduces a published reference model's rank exactly) and applied it to our Glint-protocol metrics.
+They are credible estimates, **not** confirmed board entries; an official submission is required to confirm.
 
-## Status / roadmap
+## Intended use & limitations
 
-- ✅ 16M token-scaling scan (3.2 / 6 / 10B) and 32M size baseline.
-- 🔄 16M @ expanded 8.29B corpus (16B-token "crown" run) — in progress.
-- ⏭ ARC-focused knowledge distillation (KARD) to lift ARC-Easy.
+Research artifacts for small-LM scaling studies and leaderboard work. English-only, base (not instruction-tuned)
+models at 16-32M parameters: expect limited factual knowledge and coherence. Not for production use.
 
 ## Provenance
 
-Full experiment log and artifacts in the project vault
-(`21_09_GoLLeM-v5-Skalowanie-Glint/`). Backup date: 2026-09-22.
+Full dialectical record, evaluation artifacts and methodology: labvault `21_09_GoLLeM-v5-Skalowanie-Glint/`
+(including `90-Ewaluacja/EvalHarnessParity.md` for the eval-protocol details). Trained on RunPod RTX 5090.