Correct GoLLeM-v5 rows (64M, 32M, 16M)
Browse filesThis corrects our own three rows. We found an error in our earlier evaluation script, so some numbers we submitted were too high.
**GoLLeM-v5 64M**: replaced with the final 64M model and its measurement.
- Checkpoint: `final_64m_14x576/ckpt_760k.pt` in SlayerLab/gollem-v5-ckpts, revision `3fa6f3ba` (LFS sha256 `d144c26f…`), 760,000 steps / 24.9B tokens.
- BLiMP 76.16, ARC-Easy 48.19, WikiText-2 byte perplexity 2.3474. These are our measurements with the board's `benchmark.py` protocol (BLiMP 67,000 pairs, ARC-Easy test 2,376 questions, WikiText-2 test in 256-token windows), not an official score.
- Parameters: 62.9M. The input and output embeddings are tied (one shared matrix), so it is counted once. A plain sum over the state_dict counts it twice (69.9M).
- The previous row (BLiMP 77.84, wiki 2.016, from PR #78) came from our faulty script. Compactbot's re-measurement in #78 was right on ARC-Easy (47.94) and WikiText-2 (byte ppl 2.370). Its BLiMP (75.99) used the 73,000 pairs of our old public script; on the board's 67,000 pairs that checkpoint scores 75.83.
**GoLLeM-v5 32M / 16M**: BLiMP only. The old values (73.77 / 70.53) were computed on 73,000 pairs. With the board's 67,000 pairs they are 73.48 / 70.08. ARC-Easy and WikiText-2 are unchanged.
We would be grateful if Compactbot could re-benchmark the 64M checkpoint above. Evaluation details and all checkpoints are on the model card: https://huggingface.co/SlayerLab/gollem-v5-ckpts
- index.html +7 -7
|
@@ -606,11 +606,11 @@ const models = [
|
|
| 606 |
name: "GoLLeM-v5 64M",
|
| 607 |
org: "Fabryka AI",
|
| 608 |
params: "62.9M",
|
| 609 |
-
blimp:
|
| 610 |
-
arc:
|
| 611 |
-
wiki: 2.
|
| 612 |
-
tokens: "
|
| 613 |
-
releaseDate: "2026-09-
|
| 614 |
links: {
|
| 615 |
card: "https://huggingface.co/SlayerLab/gollem-v5-ckpts"
|
| 616 |
}
|
|
@@ -619,7 +619,7 @@ const models = [
|
|
| 619 |
name: "GoLLeM-v5 32M",
|
| 620 |
org: "Fabryka AI",
|
| 621 |
params: "31.6M",
|
| 622 |
-
blimp: 73.
|
| 623 |
arc: 44.44,
|
| 624 |
wiki: 2.5386,
|
| 625 |
tokens: "16B",
|
|
@@ -632,7 +632,7 @@ const models = [
|
|
| 632 |
name: "GoLLeM-v5 16M",
|
| 633 |
org: "Fabryka AI",
|
| 634 |
params: "17.4M",
|
| 635 |
-
blimp: 70.
|
| 636 |
arc: 40.91,
|
| 637 |
wiki: 2.6746,
|
| 638 |
tokens: "16B",
|
|
|
|
| 606 |
name: "GoLLeM-v5 64M",
|
| 607 |
org: "Fabryka AI",
|
| 608 |
params: "62.9M",
|
| 609 |
+
blimp: 76.16,
|
| 610 |
+
arc: 48.19,
|
| 611 |
+
wiki: 2.3474,
|
| 612 |
+
tokens: "24.9B",
|
| 613 |
+
releaseDate: "2026-09-27",
|
| 614 |
links: {
|
| 615 |
card: "https://huggingface.co/SlayerLab/gollem-v5-ckpts"
|
| 616 |
}
|
|
|
|
| 619 |
name: "GoLLeM-v5 32M",
|
| 620 |
org: "Fabryka AI",
|
| 621 |
params: "31.6M",
|
| 622 |
+
blimp: 73.48,
|
| 623 |
arc: 44.44,
|
| 624 |
wiki: 2.5386,
|
| 625 |
tokens: "16B",
|
|
|
|
| 632 |
name: "GoLLeM-v5 16M",
|
| 633 |
org: "Fabryka AI",
|
| 634 |
params: "17.4M",
|
| 635 |
+
blimp: 70.08,
|
| 636 |
arc: 40.91,
|
| 637 |
wiki: 2.6746,
|
| 638 |
tokens: "16B",
|