---
license: apache-2.0
library_name: transformers
tags: [looped-transformer, mixture-of-experts, scaling-laws]
---

# looped-16x2-32L-d640-1e18-a100

32-layer compute-optimal checkpoint for *Sparse Layers are Critical to Scaling Looped
Language Models* ([arXiv:2605.09165](https://arxiv.org/abs/2605.09165)).

| | |
|---|---|
| architecture | `looped` |
| d_model | 640 |
| effective layers | 32 |
| loop shape | 16 layers x 2 passes |
| compute budget | 1e18 FLOPs |
| training hardware | A100-80GB |
| training steps | 45,632 |
| parameters (stored) | 143,627,520 |
| peak LR | 0.005 |
| batch size | 16 |
| muP width_ratio | 2.5 (d_base=256) |
| final val loss | n/a |

This width is the architecture's own measured A100 minimum on the 1e18 isoFLOP sweep.
Architectures optimise at different widths; at fixed compute a wider model simply trains
on fewer tokens (C = 6·N_act·D), so this is the best that architecture does with the
budget. Do not compare these against the B200 repos (`base-32L-d512-1e18`,
`looped-16x2-d640-1e18`, `looped-moe-16x2-d512-1e18`) — same widths, different hardware.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "ml-ryanlee/looped-16x2-32L-d640-1e18-a100", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("gpt2")
```

Pass `max_length=1024` when evaluating — the RoPE buffer is sized to the training context.