---
license: apache-2.0
language: [en]
tags: [deep-equilibrium, weight-tying, babylm, equilibrium-model, kinetic-ai]
datasets: [BabyLM-community/BabyLM-2026-Strict]
---

# kinetic-eqlm-anytime-121m-babylm

A weight-tied transformer block solved to a fixed point, trained with anytime
unrolled supervision. Its depth is a stopping criterion rather than an
architecture: the same 120.7M parameters are applied repeatedly until the
representation stops moving, instead of being spread across distinct layers.

## What it is for

This checkpoint exists to answer one question at a like-for-like setting: can a
model whose depth is an equilibrium computation match a conventional stack of
the same size trained on the same data for the same number of steps?

| | this model | explicit baseline |
|---|---|---|
| parameters | 120.7M | 123.8M |
| depth | one tied block, iterated | 12 distinct layers |
| BLiMP ratio vs baseline | **0.991** (3 seeds: 0.971 / 1.033 / 0.970) | 1.000 |

The mean ratio of 0.991 is statistical parity, and one seed exceeds its baseline.
Parity is the honest description; this model is not better than a conventional
transformer of the same size, it is equal to one while storing its depth as
iteration rather than as parameters.

## Training

Anytime-unrolled supervision: cross-entropy is applied to intermediate iterates
at depths 4, 8 and 12 rather than only at the final one, which makes truncated
solves usable and closes the width gap that earlier fixed-point training left
open. BabyLM strict-small, matched data, steps and batch against the baseline.

## Honest limitations

Training supervises plain unrolled iterates while evaluation budgets cap an
Anderson-accelerated solver, so the two use different algorithms at the same
nominal depth; the parity figure is a claim about the end-to-end configuration
rather than about the solver in isolation. Converting a *pretrained* large model
to this topology is a different matter and does not work at budgets we could
afford: at 1.7B, conversion retained 0.354 of above-chance benchmark capability
and grade-school arithmetic collapsed from 0.457 to 0.017 while perplexity moved
only to 1.19x baseline. Train in this topology rather than converting into it.

## Provenance

Every number traces to a results file with a resolved-config hash and commit in
https://github.com/SharathSPhD/game-llm — findings F16 through F24 for this
model, F25/F26/F35 for the conversion results.

## Loading

```python
import torch
blob = torch.load("pytorch_model.pt", map_location="cpu", weights_only=True)
# blob["config_dict"] -> EqLMConfig fields; blob["model_class"] == "EqLM"
```