---
license: apache-2.0
datasets:
  - HuggingFaceFW/fineweb
language:
  - en
library_name: pytorch
tags:
  - looped-transformer
  - weight-tying
  - recurrent-depth
  - small-language-model
---

# SimLoop 1+loop x1+1 (10.0M)

A **looped** (weight-tied recurrent) Qwen3-style transformer trained from
scratch on FineWeb under a hard budget of **<= 10M parameters** and
**<= 100M training tokens**.

One middle block is applied **K = 1** times; the layers around it are
ordinary unlooped layers:

```
embed -> [1 layer] -> ( 1 looped layer ) x K -> [1 layer] -> RMSNorm -> tied head
```

| | |
|---|---|
| parameters | **9,962,496** (incl. embeddings, input/output tied) |
| d_model / d_mlp | 384 / 720 |
| heads (GQA) | 6 query / 2 key-value, head_dim 64 |
| context | 512 tokens |
| vocabulary | 16,384 byte-level BPE trained on FineWeb |
| training tokens | 60,014,592 |
| primitives | RMSNorm, RoPE, SwiGLU, GQA, QK-norm |

## Results

| k | CE (nats) | perplexity | bits-per-byte |
|---|---|---|---|
| 0 | 5.4088 | 223.35 | 1.8913 |
| 1 | 4.3966 | 81.17 | 1.5374 **<- best** |
| 2 | 4.5995 | 99.43 | 1.6083 |
| 3 | 4.9413 | 139.95 | 1.7278 |
| 4 | 5.2659 | 193.62 | 1.8413 |
| 5 | 5.5476 | 256.63 | 1.9399 |
| 6 | 5.7900 | 327.01 | 2.0246 |
| 7 | 6.0000 | 403.41 | 2.0980 |
| 8 | 6.1835 | 484.69 | 2.1622 |
| 9 | 6.3452 | 569.77 | 2.2188 |
| 10 | 6.4887 | 657.65 | 2.2689 |
| 11 | 6.6166 | 747.41 | 2.3137 |
| 12 | 6.7313 | 838.20 | 2.3537 |

Reference points on the same validation split: a context-free **unigram** model scores CE 7.5476 (ppl 1896.10); **uniform** over the vocabulary scores CE 9.7041.


`k` is the number of applications of the looped block at inference. The model
is weight-tied, so **any k can be run**; the table is a single checkpoint
evaluated at every depth.

**Measured behaviour:** TRAINED UNLOOPED (K=1): the k>1 rows show how a model trained at one application behaves when looped anyway, not whether looping pays.

- first application of the looped block buys **+1.0122** nats (k=0 -> k=1)
- all further applications buy **+0.0000** nats (k=1 -> k=1)


## Usage

```python
import torch
from simloop.stack import StackConfig, StackedLoop

ck = torch.load("model.pt", map_location="cpu", weights_only=False)
model = StackedLoop(StackConfig(**ck["model_cfg"]))
model.load_state_dict(ck["model"]); model.eval()

ids = torch.tensor([[1, 2, 3]])            # from tokenizer.json
logits = model(ids, K=1)           # try other K: the block is tied
```

See `load_example.py`. The tokenizer is a `tokenizers` BPE:
`Tokenizer.from_file("tokenizer.json")`.

## Honest limitations

- Trained on 60,014,592 tokens at ~10M parameters. It is a research artifact
  for studying looped depth, **not** a useful general-purpose language model.
- Perplexity is tokenizer-dependent; **bits-per-byte** is the comparable
  number and is reported above.
- The looped block **saturates**: past the depth listed as best above, extra
  applications make cross-entropy worse, not better. This is measured, not
  assumed, and is the central finding of the project.
- English-only, no instruction tuning, no safety filtering beyond FineWeb's.

Full experimental record, including every failed experiment:
https://github.com/brkdrd/SimLoop