File size: 2,648 Bytes
6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 6010979 4a194a5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
license: apache-2.0
language: [en]
tags: [deep-equilibrium, weight-tying, babylm, equilibrium-model, kinetic-ai]
datasets: [BabyLM-community/BabyLM-2026-Strict]
---
# kinetic-eqlm-anytime-121m-babylm
A weight-tied transformer block solved to a fixed point, trained with anytime
unrolled supervision. Its depth is a stopping criterion rather than an
architecture: the same 120.7M parameters are applied repeatedly until the
representation stops moving, instead of being spread across distinct layers.
## What it is for
This checkpoint exists to answer one question at a like-for-like setting: can a
model whose depth is an equilibrium computation match a conventional stack of
the same size trained on the same data for the same number of steps?
| | this model | explicit baseline |
|---|---|---|
| parameters | 120.7M | 123.8M |
| depth | one tied block, iterated | 12 distinct layers |
| BLiMP ratio vs baseline | **0.991** (3 seeds: 0.971 / 1.033 / 0.970) | 1.000 |
The mean ratio of 0.991 is statistical parity, and one seed exceeds its baseline.
Parity is the honest description; this model is not better than a conventional
transformer of the same size, it is equal to one while storing its depth as
iteration rather than as parameters.
## Training
Anytime-unrolled supervision: cross-entropy is applied to intermediate iterates
at depths 4, 8 and 12 rather than only at the final one, which makes truncated
solves usable and closes the width gap that earlier fixed-point training left
open. BabyLM strict-small, matched data, steps and batch against the baseline.
## Honest limitations
Training supervises plain unrolled iterates while evaluation budgets cap an
Anderson-accelerated solver, so the two use different algorithms at the same
nominal depth; the parity figure is a claim about the end-to-end configuration
rather than about the solver in isolation. Converting a *pretrained* large model
to this topology is a different matter and does not work at budgets we could
afford: at 1.7B, conversion retained 0.354 of above-chance benchmark capability
and grade-school arithmetic collapsed from 0.457 to 0.017 while perplexity moved
only to 1.19x baseline. Train in this topology rather than converting into it.
## Provenance
Every number traces to a results file with a resolved-config hash and commit in
https://github.com/SharathSPhD/game-llm — findings F16 through F24 for this
model, F25/F26/F35 for the conversion results.
## Loading
```python
import torch
blob = torch.load("pytorch_model.pt", map_location="cpu", weights_only=True)
# blob["config_dict"] -> EqLMConfig fields; blob["model_class"] == "EqLM"
```
|