Spaces:

---

AxiomicLabs

/

Open_SLM_Leaderboard

like 88

Running

Adding cagliostro-v2, a 150M model we pretrained from scratch on 50B tokens.

https://huggingface.co/bench-labs/cagliostro-v2

Zero-shot acc_norm from lm-eval, ArithMark from bencharithmark-3.py at its default batch size of 32:

We ran the full eval twice, on different GPUs and at different batch sizes. The second run came out at 19.93, so 19.99 is the mean if you would rather use an averaged figure. Raw lm-eval output files are available if you want them for verification.

Params are 150,038,400 with tied embeddings. That is slightly over the 150M line in Rule 01, same as a few entries already on the board including our own v1 at 157.4M, so hopefully still in scope.

Base checkpoint, not merged, not instruction tuned. Architecture is a 30 layer 640 dim decoder with grouped-query attention, plus Exclusive Self Attention (arXiv:2603.09078) and logit soft-capping at 15. Both add zero parameters. Loading needs trust_remote_code=True since the architecture is not in transformers.

This is a separate model from our cagliostro-v1 entry, not a replacement.