Upload README.md with huggingface_hub

README.md

@@ -1,25 +1,109 @@
-# GoLLeM-v5 — modele (backup lokalny)
+---
+license: apache-2.0
+language:
+- en
+tags:
+- tiny-lm
+- babylm
+- language-model
+- from-scratch
+- gpt
+library_name: pytorch
+pipeline_tag: text-generation
+---
 
-Backup ckptów treningowych GoLLeM-v5 (scan tokenów 16M). Źródło: RunPod RTX 5090.
-Redundancja: pod (oryginał) + HF `SlayerLab/minimal-en-corpus-5b:ckpts/` (durable) + ten katalog (lokalny).
+# GoLLeM-v5 — English Tiny-LM Checkpoints
 
-## Modele (scan tokenów, single-factor, BPE-12k, L6/d408/h6, block1024, batch64, AdamW lr6e-4→6e-5, seed1337)
+Training checkpoints for **GoLLeM-v5**, a family of small English language models
+(≈17M–31M parameters) trained **from scratch** on a curated, decontaminated English
+corpus. Built for the **sub-100M efficiency regime** and evaluated on the tiny-ML
+leaderboard tasks: **BLiMP**, **ARC-Easy**, and **WikiText-2**.
 
-| plik | tokeny | epoki | BLiMP | ARC-Easy | WikiText-2 BPB | board-pozycja |
-|------|--------|-------|-------|----------|----------------|---------------|
-| run_bpe16m_c_ckpt.pt    | 3.2B | 1.19 | 71.54 | 39.39 | 1.2161 | #16/74 |
-| run_bpe16m_6b_d_ckpt.pt | 6B   | 2.2  | 73.29 | 41.04 | 1.1943 | #11/74 |
-| run_bpe16m_10b_e_ckpt.pt| 10B  | 3.7  | 73.43 | 41.67 | 1.1815 | (liczy Latarnik) |
+Trained on a single RunPod **RTX 5090**. Redundant backups: this HF repo (durable) +
+the training pod (origin) + local disk.
 
-**Wniosek scanu:** BLiMP saturuje ~6B na 16M (Δ6→10=+0.14); ARC dalej rośnie @10B (knee>10B); BPB monotoniczny.
+## Architecture
 
-- `tokenizer.json` — BPE-12k (vocab 12288), ZALOCKOWANY (werdykt 3/3 vs byte-baseline: BLiMP+2σ ∧ BPB-Δ0.238 ∧ board-#16). Wspólny dla wszystkich modeli v5.
-- Config eval: `byte_lm_eval_bpe.py --tokenizer tokenizer.json --ckpt <model> --tasks wikitext,blimp,arc_easy --n-head 6`.
-- Korpus treningowy: `SlayerLab/minimal-en-corpus-5b` (2.70B unique BPE-tok, FineWeb-Edu, ARC-targeted, decontam'd).
+All 16M checkpoints share one decoder-only GPT (`GPT-ref`):
 
-## Kolejne (dojdą po treningu)
-- 32M-baseline (L6/d576/h9, izolacja-rozmiaru) + lewary single-factor (RoPE/low-rank/Muon/depth).
-- 150M flagowiec (+KARD-distill z Qwena).
+| | |
+|---|---|
+| Layers / width / heads | **6 / 408 / 6** (≈17.4M params) |
+| Context length | 1024 |
+| Vocabulary | 12288 (BPE-12k) |
+| Optimizer | AdamW, lr 6e-4 → 6e-5 (cosine) |
+| Precision / seed | bf16 / 1337 |
 
-Prowieniencja pełna: labvault `21_09_GoLLeM-v5-Skalowanie-Glint/` (Stan.md + artifacts + S2-Wnioski).
-Data backupu: 2026-09-22.
+The 32M baseline is the same recipe at **L6 / d576 / h9 (≈31.4M params)**.
+
+## Checkpoints
+
+### Token-scaling scan (16M, single-factor: identical arch/hypers, only the token budget changes)
+
+| file | tokens | epochs | BLiMP | ARC-Easy | WikiText-2 (BPB) |
+|------|-------:|-------:|------:|---------:|-----------------:|
+| `bpe16m_3.2B/ckpt.pt` | 3.2B | 1.2 | 71.54 | 39.39 | 1.2161 |
+| `bpe16m_6B/ckpt.pt`   | 6B   | 2.2 | 73.29 | 41.04 | 1.1943 |
+| `bpe16m_10B/ckpt.pt`  | 10B  | 3.7 | 73.43 | 41.67 | 1.1815 |
+
+Metrics above are from `lm-evaluation-harness` (BLiMP full, ARC-Easy, WikiText-2 bits-per-byte).
+
+**Finding:** BLiMP saturates around ~6B tokens for this 16M model (Δ 6→10B = +0.14);
+ARC-Easy keeps improving at 10B; BPB decreases monotonically.
+
+### 32M size-isolation baseline
+
+| file | params | tokens |
+|------|-------:|-------:|
+| `bpe32m_baseline/ckpt.pt` | 31.4M (L6/d576/h9) | 10B |
+
+### Leaderboard protocol (Glint) results
+
+The public leaderboard uses a stricter evaluation protocol (256-token clip, no BOS,
+train-split BLiMP, bare-prompt ARC-Easy raw accuracy, byte-normalized WikiText). Under
+that protocol our numbers are lower than the `lm-eval` numbers above (offset ≈ 3.1pp
+BLiMP / 2.2pp ARC) — **always cite the Glint numbers for the board**:
+
+| model | tokens | BLiMP | ARC-Easy | efficiency rank |
+|-------|-------:|------:|---------:|:---------------:|
+| 16M | 10B | 70.36 | 39.52 | #18 / 74 |
+| 32M | 10B | 70.08 | 42.59 | #20 / 74 |
+
+The 32M model ranks *below* the 16M model on efficiency: the leaderboard's size bonus
+favors smaller models, and the extra ARC gain does not offset the reduced bonus — hence
+the crown effort stays at the 16M size.
+
+## Tokenizer
+
+`tokenizer.json` — BPE, vocab 12288 ("BPE-12k"), shared by every v5 model. Locked after a
+3/3 verdict against a byte-level baseline (BLiMP +2σ ∧ WikiText-2 BPB Δ0.238 ∧ improved
+leaderboard efficiency).
+
+## Training data
+
+- Base corpus: **`SlayerLab/minimal-en-corpus-5b`** — 5.40B unique BPE-12k tokens, a
+  15-source English mixture (FineWeb-Edu, DCLM, code, math, books, QA, science),
+  decontaminated against the benchmark test sets.
+- An **expanded 8.29B-token** corpus (base + additional FineWeb-Edu-100BT + OpenStax
+  science, all deduplicated and decontaminated) backs the in-progress 16M crown run.
+
+## Usage
+
+```python
+import torch
+ckpt = torch.load("bpe16m_10B/ckpt.pt", map_location="cpu")
+sd = ckpt["model"] if "model" in ckpt else ckpt
+# GPT-ref: L6 d408 h6, vocab 12288, block 1024.
+# Model class + forward: train_gpt_ref.py. Trim logits to vocab 12288 (no padded vocab).
+```
+
+## Status / roadmap
+
+- ✅ 16M token-scaling scan (3.2 / 6 / 10B) and 32M size baseline.
+- 🔄 16M @ expanded 8.29B corpus (16B-token "crown" run) — in progress.
+- ⏭ ARC-focused knowledge distillation (KARD) to lift ARC-Easy.
+
+## Provenance
+
+Full experiment log and artifacts in the project vault
+(`21_09_GoLLeM-v5-Skalowanie-Glint/`). Backup date: 2026-09-22.