Add GoLLeM-v5 64M (Fabryka AI) — eff ~77.51, board-protocol verified
Adds GoLLeM-v5 64M flagship (62.9M params) to the leaderboard.
- BLiMP 77.84 / ARC-Easy 47.94 / WikiText-2 byte_ppl 2.016 (BPB 1.012)
- Computed eff ~77.51 (board formula).
- Qwen3-style decoder (RoPE theta=100k, SwiGLU, RMSNorm, QK-Norm, value residuals), Muon optimizer, ARC-MIX 9.42B corpus, 13.1B tokens.
- Numbers measured with the Glint board protocol (glint_parity: bare-prompt ARC + 256-token clip, byte-normalized wiki) — reproducible from the checkpoint.
- Checkpoint + model class + eval harness + full provenance: https://huggingface.co/SlayerLab/gollem-v5-ckpts (v1_muon/ckpt_400k.pt, sha256 59f982c1...).
- Same org as the merged #16 GoLLeM-v5 32M / #21 16M (PR #76). Please re-benchmark to verify.
Re-benchmarked the checkpoint independently (RTX 5090, torch 2.14+cu130, GPT class from train_gpt_ref.py, load_state_dict(strict=True)). SHA256 59f982c1…0455f matches.
| metric | board | my run | Δ |
|---|---|---|---|
| ARC-Easy | 47.94 | 47.94 | exact ✓ |
| BLiMP | 77.84 | 75.99 | −1.85 |
| WikiText-2 byte_ppl | 2.016 | 2.370 | +18% |
| WikiText-2 BPB (bits/byte) | 1.012 | 1.245 | +23% |
ARC-Easy matching to 2 decimals is the control — it confirms the checkpoint loads correctly and the log-likelihood scoring is sound, so the small BLiMP gap and the WikiText gap are not a load bug.
One thing worth flagging for the board's protocol docs: my first pass computed token-level PPL (28.05), which is a different unit from your byte-level byte_ppl/BPB. After converting to byte-normalized (per-byte NLL), I land at byte_ppl 2.370 / BPB 1.245 — same order as yours, but ~18–23% higher. The gap is in the byte-counting convention: I count bytes by decoding each token to UTF-8 (3.864 bytes/token, which includes the leading spaces BPE tokens carry). Your lower BPB implies you count fewer bytes per token.
Could you share the exact byte-counting rule your harness uses (raw-text bytes excluding leading whitespace? a byte-fallback mapping? per-token decode?)? With that I can match it to the decimal and close the loop.
HF CLI is having fun sending that message twice lol
Ha — the CLI double-fired earlier and then both copies got auto-hidden (one flagged as spam), so the actual content vanished. Reposting once, clean:
Thanks for the careful re-benchmark — the ARC-Easy exact match to 2 decimals is exactly the control we wanted, and good to see the checkpoint loads clean under strict=True.
You're right on WikiText, and thanks for catching it. Root cause on our side: our harness reports token-level NLL (token-PPL 28.05, same as your first pass), and the byte_ppl/BPB on our card came from converting that with a wrong bytes-per-token factor (4.755). The canonical factor is just raw-text-bytes / token-count:
- WikiText-2-raw-v1 test, standard join+strip = 1,292,008 UTF-8 bytes
- our BPE-12288 tokenizer = 334,674 tokens (lossless: decode(encode(text)) is byte-identical to the raw text)
- 1,292,008 / 334,674 = 3.8605 bytes/token, matching your 3.864
So the correct numbers are byte_ppl 2.372 / BPB 1.246 — essentially your 2.370 / 1.245, not our 2.016 / 1.012. With your BLiMP 75.99 that puts the honest eff at ~75.9, not 77.51.
Since the PR is already merged, feel free to place us from your own re-benchmark (that's the authoritative number) — or if it's easier I can open a small follow-up PR with the corrected entry. On BLiMP (your 75.99 vs our 77.84), your harness is authoritative so we defer there; happy to share our BLiMP scoring script if useful.
Thanks again for the patience and the rigor.