---
license: apache-2.0
language: en
library_name: pytorch
tags:
- interpretability
- geometric-algebra
- operator-language-model
- reproducibility
datasets:
- HuggingFaceFW/fineweb
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0
- HuggingFaceTB/cosmopedia
- HuggingFaceTB/smoltalk
- Salesforce/wikitext
---

# cl33-opLM — selective reproducibility release (preprint v1.1)

Paper: **One Object: Memory, Navigation, and Reportability in an Operator-Only
Language Model** — DOI [10.5281/zenodo.22684392](https://doi.org/10.5281/zenodo.22684392)
· https://t3atlas.dev/cl33/paper/ · Live demo: https://cl33.t3atlas.dev
Author: Garret Sutherland, MirrorEthic LLC.

This bundle contains what is necessary to **independently test the published claims**
on frozen artifacts. It is deliberately not the training stack: the paper's §12
program is ongoing and its machinery is not included. Reproducibility surface ≠
complete source disclosure.

## Contents

| file | what | sha256 |
|---|---|---|
| `cl33_oplm_prose_236m.pt` | prose base, 236.5M, step 189307 (Table 1b checkpoint) | `fe407328…6a1fbf` |
| `cl33_oplm_chat_236m.pt`  | chat/serving model, step 13996 (the cl33.t3atlas.dev model) | `ddd042a6…559535` |
| `invert_probe_chat.pt`    | reverse-readout probe (held-out top-1 0.860, card inside) | `91201342…ca79ef` |
| `model_v2.py` + `model.py` + `so33.py` + `wedge.py` + `t3v3_wedge_memory.py` + `tape_memory.py` | model definition (load-only) | — |
| `repro_bottleneck.py`     | Claim 1: the mandatory operator bottleneck | — |
| `repro_reverse_readout.py`| Claim 2: the operator stream is a transcript | — |
| `SHA256SUMS`              | full hashes | — |

Full hashes in `SHA256SUMS`. Deps: `torch`, `transformers`, `datasets` (Python ≥3.10).

## Claim 1 — the operator bottleneck is mandatory

Zero the emitted operators; the model loses its only path to output.

```
# EXACT in-domain reproduction (frozen 24x1025-token slice of the prose-mix val, shipped):
python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt --slice eval_slice_prose_val.npy
# off-domain, public data:
python repro_bottleneck.py --ckpt cl33_oplm_prose_236m.pt
python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt
```

Expected (WikiText-103 test, public data, seq 1024 — measured on this exact bundle):

| ckpt | native PPL | ops-off PPL | ratio |
|---|---|---|---|
| chat + frozen slice (exact) | 68.6 | 18,552.6 | **270×** |
| prose + wikitext | ≈61 | ≈6,500 | **≈106×** |
| chat + wikitext  | ≈186 | ≈21,000 | **≈112×** |

(The paper's original 314× was a different random draw of the same validation mix;
the shipped frozen slice reproduces **exactly** at 270×, and the claim — orders of
magnitude — holds on public data too. `eval_slice_prose_val.npy` is derived data
(GPT-2 BPE token ids) drawn from public corpora: FineWeb/FineWeb-Edu (ODC-By), DCLM,
Project Gutenberg (public domain), Wikipedia (CC BY-SA), Cosmopedia (Apache-2.0),
FineMath, Stack-Edu; shipped solely as an evaluation fixture with attribution.)

## Claim 2 — the operator stream is a readable transcript

A probe that sees ONLY the emitted operators (no token input) decodes the text:

```
python repro_reverse_readout.py --text "any sentence you like"
```

Expected: ~0.86 top-1 on typical English (the probe's held-out card prints on load;
rare words fail toward semantic neighbors — that is the paper's §7 claim, not a bug).


## Training data statement

Four stages, ~11.8B tokens cumulative (GPT-2 BPE; chat stages use a 5-token spliced
extension → vocab 50262). `cl33_oplm_prose_236m.pt` is the Stage-2 endpoint;
`cl33_oplm_chat_236m.pt` (the cl33.t3atlas.dev demo model) is the Stage-4 endpoint.

**Stage 1 — from-scratch pretrain, 5.000B tokens ("ultimate_mix"):**
FineWeb-Edu 2.00B (40%) · DCLM 1.00B (20%) · Cosmopedia 0.50B (10%) · FineMath 0.50B
(10%) · Stack-Edu 0.50B code (10%) · Wikipedia 0.50B (10%). All 56 shards
sha1-fingerprinted in the project's run manifest.

**Stage 2 — context splice 512→1024 + prose continuation, 6.20B tokens ("prose_mix"):**
FineWeb 35% · Gutenberg 15% · Wikipedia 10% · DCLM 10% · FineWeb-Edu 10% · Cosmopedia
8% · FineMath 6% · Stack-Edu 6%. This is the checkpoint whose bits-per-byte matches
token-matched Pythia-160m (paper, Table 1b).

**Stage 3 — chat SFT, ~0.43B tokens, 2 epochs:** 327k chat conversations
(SmolTalk-derived + a deduplicated diverse set + 5,472 in-house self-Q&A pairs + 120
persona seeds), 61k self-corpus document chunks, 10k pretrain-replay documents
(forgetting guard), 427 reasoning traces (287 R1-derived CoT + 140 general;
system-prompt-gated).

**Stage 4 — pinned-persona SFT, ~0.19B tokens:** identity trained as a pinned tape
record (authority channel) rather than system-prompt tokens; in-house synthetic persona
corpus.

Sources are public corpora (FineWeb/FineWeb-Edu ODC-By; DCLM CC-BY-4.0; Cosmopedia
Apache-2.0; Wikipedia CC-BY-SA; Gutenberg public domain; SmolTalk Apache-2.0; Stack-Edu
per-repository licenses; FineMath ODC-By) plus in-house synthetic material (self-Q&A,
persona, general-think; the 287 CoT traces are R1-distilled). No private or user data
in any stage. The shipped `eval_slice_prose_val.npy` is a 24×1025-token fixture drawn
from the Stage-2 validation split.

## What is NOT here, and why

Training orchestration, data pipelines, the §12 memory-organ program (labeled ongoing
in the paper), and downstream control/steering machinery. The claims those support are
either reported with their own dated work-log provenance (paper, Appendix R) or not
yet published. This bundle is scoped to verify what the preprint asserts about these
frozen artifacts.

## Checksums / provenance

Both checkpoints are weights-only exports (optimizer state stripped) of the exact
training checkpoints named in the paper's Appendix R. Verify with:
`sha256sum -c SHA256SUMS`