kinetic-eqlm-anytime-121m-babylm

A weight-tied transformer block solved to a fixed point, trained with anytime unrolled supervision. Its depth is a stopping criterion rather than an architecture: the same 120.7M parameters are applied repeatedly until the representation stops moving, instead of being spread across distinct layers.

What it is for

This checkpoint exists to answer one question at a like-for-like setting: can a model whose depth is an equilibrium computation match a conventional stack of the same size trained on the same data for the same number of steps?

	this model	explicit baseline
parameters	120.7M	123.8M
depth	one tied block, iterated	12 distinct layers
BLiMP ratio vs baseline	0.991 (3 seeds: 0.971 / 1.033 / 0.970)	1.000

The mean ratio of 0.991 is statistical parity, and one seed exceeds its baseline. Parity is the honest description; this model is not better than a conventional transformer of the same size, it is equal to one while storing its depth as iteration rather than as parameters.

Training

Anytime-unrolled supervision: cross-entropy is applied to intermediate iterates at depths 4, 8 and 12 rather than only at the final one, which makes truncated solves usable and closes the width gap that earlier fixed-point training left open. BabyLM strict-small, matched data, steps and batch against the baseline.

Honest limitations

Training supervises plain unrolled iterates while evaluation budgets cap an Anderson-accelerated solver, so the two use different algorithms at the same nominal depth; the parity figure is a claim about the end-to-end configuration rather than about the solver in isolation. Converting a pretrained large model to this topology is a different matter and does not work at budgets we could afford: at 1.7B, conversion retained 0.354 of above-chance benchmark capability and grade-school arithmetic collapsed from 0.457 to 0.017 while perplexity moved only to 1.19x baseline. Train in this topology rather than converting into it.

Provenance

Every number traces to a results file with a resolved-config hash and commit in https://github.com/SharathSPhD/game-llm — findings F16 through F24 for this model, F25/F26/F35 for the conversion results.

Loading

import torch
blob = torch.load("pytorch_model.pt", map_location="cpu", weights_only=True)
# blob["config_dict"] -> EqLMConfig fields; blob["model_class"] == "EqLM"