Maggio33 commited on
Commit
a820ef1
·
verified ·
1 Parent(s): a2ef270

Correct GoLLeM-v5 rows (64M, 32M, 16M)

Browse files

This corrects our own three rows. We found an error in our earlier evaluation script, so some numbers we submitted were too high.

**GoLLeM-v5 64M**: replaced with the final 64M model and its measurement.
- Checkpoint: `final_64m_14x576/ckpt_760k.pt` in SlayerLab/gollem-v5-ckpts, revision `3fa6f3ba` (LFS sha256 `d144c26f…`), 760,000 steps / 24.9B tokens.
- BLiMP 76.16, ARC-Easy 48.19, WikiText-2 byte perplexity 2.3474. These are our measurements with the board's `benchmark.py` protocol (BLiMP 67,000 pairs, ARC-Easy test 2,376 questions, WikiText-2 test in 256-token windows), not an official score.
- Parameters: 62.9M. The input and output embeddings are tied (one shared matrix), so it is counted once. A plain sum over the state_dict counts it twice (69.9M).
- The previous row (BLiMP 77.84, wiki 2.016, from PR #78) came from our faulty script. Compactbot's re-measurement in #78 was right on ARC-Easy (47.94) and WikiText-2 (byte ppl 2.370). Its BLiMP (75.99) used the 73,000 pairs of our old public script; on the board's 67,000 pairs that checkpoint scores 75.83.

**GoLLeM-v5 32M / 16M**: BLiMP only. The old values (73.77 / 70.53) were computed on 73,000 pairs. With the board's 67,000 pairs they are 73.48 / 70.08. ARC-Easy and WikiText-2 are unchanged.

We would be grateful if Compactbot could re-benchmark the 64M checkpoint above. Evaluation details and all checkpoints are on the model card: https://huggingface.co/SlayerLab/gollem-v5-ckpts

Files changed (1) hide show
  1. index.html +7 -7
index.html CHANGED
@@ -606,11 +606,11 @@ const models = [
606
  name: "GoLLeM-v5 64M",
607
  org: "Fabryka AI",
608
  params: "62.9M",
609
- blimp: 77.84,
610
- arc: 47.94,
611
- wiki: 2.016,
612
- tokens: "13.1B",
613
- releaseDate: "2026-09-24",
614
  links: {
615
  card: "https://huggingface.co/SlayerLab/gollem-v5-ckpts"
616
  }
@@ -619,7 +619,7 @@ const models = [
619
  name: "GoLLeM-v5 32M",
620
  org: "Fabryka AI",
621
  params: "31.6M",
622
- blimp: 73.77,
623
  arc: 44.44,
624
  wiki: 2.5386,
625
  tokens: "16B",
@@ -632,7 +632,7 @@ const models = [
632
  name: "GoLLeM-v5 16M",
633
  org: "Fabryka AI",
634
  params: "17.4M",
635
- blimp: 70.53,
636
  arc: 40.91,
637
  wiki: 2.6746,
638
  tokens: "16B",
 
606
  name: "GoLLeM-v5 64M",
607
  org: "Fabryka AI",
608
  params: "62.9M",
609
+ blimp: 76.16,
610
+ arc: 48.19,
611
+ wiki: 2.3474,
612
+ tokens: "24.9B",
613
+ releaseDate: "2026-09-27",
614
  links: {
615
  card: "https://huggingface.co/SlayerLab/gollem-v5-ckpts"
616
  }
 
619
  name: "GoLLeM-v5 32M",
620
  org: "Fabryka AI",
621
  params: "31.6M",
622
+ blimp: 73.48,
623
  arc: 44.44,
624
  wiki: 2.5386,
625
  tokens: "16B",
 
632
  name: "GoLLeM-v5 16M",
633
  org: "Fabryka AI",
634
  params: "17.4M",
635
+ blimp: 70.08,
636
  arc: 40.91,
637
  wiki: 2.6746,
638
  tokens: "16B",