README.md

---
license: mit
tags:
  - latent-reasoning
  - chain-of-thought
  - coconut
  - curriculum-learning
---

# lyot-chain10 — Coconut latent reasoning on a depth-10 chain

Four GPT-2-style checkpoints (2 layers, 15.0M params, 40-token symbolic vocabulary) trained
with [Coconut](https://github.com/facebookresearch/coconut)-style continuous chain-of-thought
on a graph-reachability task requiring **ten sequential latent steps**.

Part of *Learn Your Own Thoughts* (ATC). Coconut's previous ceiling in our experiments was
depth 4 under a fixed clock schedule; these reach **eval accuracy 0.999** at depth 10.

## Task

Which of two candidate nodes is reachable from the root?

```
<eos> 3 17 | 8 1 | 15 16 | ... | 5 7 [Q] 19 13 [R] 20 [A]  ->  19
      \___ edges ___/               \cands/  \root/
```

A depth-10 chain plus an **isomorphic unreachable twin** — 22 nodes, 20 edges, 77 tokens. The
distractor is the target's counterpart in the twin component, matching depth and in/out-degree,
so no local feature separates the candidates. Chance is 0.500. Node ids are permuted per task,
which removes an id-ordering shortcut worth ~0.30 accuracy.

## Checkpoints

Each is a full training state: `model`, `optimizer`, `epoch`, `stage`, `metrics`.

| file | uniform_prob | shuffle_nodes | epoch | eval/acc |
|---|---|---|---|---|
| `...ucou7ar1-ep157-s10-eval0.999.pt` | 0.5 | true | 157 | **0.999** |
| `...bjmzm8ql-ep139-s10-eval0.998.pt` | 0.1 | false | 139 | **0.998** |
| `...pclc3j4q-ep116-s10-eval0.998.pt` | 0.5 | false | 116 | **0.998** |
| `...6oth0ni8-ep162-s10-eval0.964.pt` | 0.1 | true | 162 | 0.964 |

`eval/acc` is on 1000 held-out graphs at full difficulty and full latent budget. The filename
embeds the wandb run id (project `lyot-chain10`). All four were at curriculum stage 10 when
saved. The best is `shuffle_nodes=true`, the shortcut-free setting.

### The clock baseline (`c10clk-*`), at chance

The four `c10clk-*` files are the control, and they are **chance-level by design, not
broken**. Everything about them matches the arms above except that the curriculum advances
on upstream's fixed schedule (`epochs_per_stage: 5`) rather than on measured accuracy:

| file | uniform_prob | shuffle_nodes | epoch | eval/acc |
|---|---|---|---|---|
| `c10clk-...fpncimyu-ep162-s10-eval0.510.pt` | 0.5 | false | 162 | 0.510 |
| `c10clk-...3yqogp0e-ep161-s10-eval0.509.pt` | 0.1 | false | 161 | 0.509 |
| `c10clk-...zkz6ansr-ep158-s10-eval0.496.pt` | 0.1 | true | 158 | 0.496 |
| `c10clk-...1vleof96-ep160-s10-eval0.488.pt` | 0.5 | true | 160 | 0.488 |

Chance is 0.500. All four reached stage 10 by epoch 50 — three times sooner than the adaptive
arms — and none exceeded 0.537 at any point in ~160 epochs. They are published because the
depth-10 claim rests on this comparison: same task, same model, same `reset_optimizer=true`,
differing only in how stages advance.

## What made depth 10 reachable

Two changes over the standard recipe, from a 2x2x2x2 factorial plus this baseline:

1. **`reset_optimizer=true`** — rebuild AdamW every epoch, discarding both moment estimates.
   Necessary: all four adaptive arms with it reached stage 10 and >0.96 eval; **no arm
   without it passed stage 7 or 0.07 eval**, given ~2x the epochs. The curriculum keeps
   shifting the input distribution, and stale second moments appear to misscale the new
   stage's gradients.
2. **Accuracy-driven staging** (`staging: revisit`, threshold 0.9, patience 5) — advance only
   after a stage passes 5 consecutive epochs at its own budget, and fall back to any earlier
   stage that regresses.

Neither alone is enough. The `c10clk-*` arms have the optimizer reset and still sit at chance,
so the reset is necessary but not sufficient; and no arm without the reset got near stage 10
under either schedule. Arriving at the final stage early is worth nothing if the stages were
never learned on the way.

`uniform_prob` and `shuffle_nodes` only modulate how fast the schedule progresses; neither
rescues a run without `reset_optimizer`.

## Caveats

- **Snapshots of a live oscillation, not converged models.** The revisit scheduler repeatedly
  drops back to an earlier stage, and eval swings between ~0.3 and 1.000 before recovering.
  These are caught at stage 10; they are not best-ever weights, and no LR decay was used.
- **Depth 10, not depth 10 held.** The 1.000 recurs across many epochs rather than once, but
  "solves chain-10" describes a peak becoming a plateau.
- **2 layers beats 4.** In the same factorial no 4-layer arm passed stage 7 (best eval 0.17).
- Trained on a synthetic 40-token symbolic vocabulary; not a language model.

## Loading

```python
import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("seyedparsa/lyot-chain10",
                       "c10-L2-u0.5-rtrue-strue-lyot-chain10_ucou7ar1-ep157-s10-eval0.999.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
print(ckpt["epoch"], ckpt["stage"], ckpt["metrics"]["eval/acc"])
state = ckpt["model"]
```

Config: `n_layer=2, n_head=8, n_embd=768, n_positions=1024`, `method=coconut`,
`curriculum=path`, `c_thought=1`, `max_latent_stage=10`, `lr=1e-4`, `batch_size=128`.