license: apache-2.0
language: en
library_name: pytorch
tags:
- interpretability
- geometric-algebra
- operator-language-model
- reproducibility
datasets:
- HuggingFaceFW/fineweb
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0
- HuggingFaceTB/cosmopedia
- HuggingFaceTB/smoltalk
- Salesforce/wikitext
cl33-opLM — selective reproducibility release (preprint v1.1)
Paper: One Object: Memory, Navigation, and Reportability in an Operator-Only Language Model — DOI 10.5281/zenodo.22684392 · https://t3atlas.dev/cl33/paper/ · Live demo: https://cl33.t3atlas.dev Author: Garret Sutherland, MirrorEthic LLC.
This bundle contains what is necessary to independently test the published claims on frozen artifacts. It is deliberately not the training stack: the paper's §12 program is ongoing and its machinery is not included. Reproducibility surface ≠ complete source disclosure.
Contents
| file | what | sha256 |
|---|---|---|
cl33_oplm_prose_236m.pt |
prose base, 236.5M, step 189307 (Table 1b checkpoint) | fe407328…6a1fbf |
cl33_oplm_chat_236m.pt |
chat/serving model, step 13996 (the cl33.t3atlas.dev model) | ddd042a6…559535 |
invert_probe_chat.pt |
reverse-readout probe (held-out top-1 0.860, card inside) | 91201342…ca79ef |
model_v2.py + model.py + so33.py + wedge.py + t3v3_wedge_memory.py + tape_memory.py |
model definition (load-only) | — |
repro_bottleneck.py |
Claim 1: the mandatory operator bottleneck | — |
repro_reverse_readout.py |
Claim 2: the operator stream is a transcript | — |
SHA256SUMS |
full hashes | — |
Full hashes in SHA256SUMS. Deps: torch, transformers, datasets (Python ≥3.10).
Claim 1 — the operator bottleneck is mandatory
Zero the emitted operators; the model loses its only path to output.
# EXACT in-domain reproduction (frozen 24x1025-token slice of the prose-mix val, shipped):
python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt --slice eval_slice_prose_val.npy
# off-domain, public data:
python repro_bottleneck.py --ckpt cl33_oplm_prose_236m.pt
python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt
Expected (WikiText-103 test, public data, seq 1024 — measured on this exact bundle):
| ckpt | native PPL | ops-off PPL | ratio |
|---|---|---|---|
| chat + frozen slice (exact) | 68.6 | 18,552.6 | 270× |
| prose + wikitext | ≈61 | ≈6,500 | ≈106× |
| chat + wikitext | ≈186 | ≈21,000 | ≈112× |
(The paper's original 314× was a different random draw of the same validation mix;
the shipped frozen slice reproduces exactly at 270×, and the claim — orders of
magnitude — holds on public data too. eval_slice_prose_val.npy is derived data
(GPT-2 BPE token ids) drawn from public corpora: FineWeb/FineWeb-Edu (ODC-By), DCLM,
Project Gutenberg (public domain), Wikipedia (CC BY-SA), Cosmopedia (Apache-2.0),
FineMath, Stack-Edu; shipped solely as an evaluation fixture with attribution.)
Claim 2 — the operator stream is a readable transcript
A probe that sees ONLY the emitted operators (no token input) decodes the text:
python repro_reverse_readout.py --text "any sentence you like"
Expected: ~0.86 top-1 on typical English (the probe's held-out card prints on load; rare words fail toward semantic neighbors — that is the paper's §7 claim, not a bug).
Training data statement
Four stages, ~11.8B tokens cumulative (GPT-2 BPE; chat stages use a 5-token spliced
extension → vocab 50262). cl33_oplm_prose_236m.pt is the Stage-2 endpoint;
cl33_oplm_chat_236m.pt (the cl33.t3atlas.dev demo model) is the Stage-4 endpoint.
Stage 1 — from-scratch pretrain, 5.000B tokens ("ultimate_mix"): FineWeb-Edu 2.00B (40%) · DCLM 1.00B (20%) · Cosmopedia 0.50B (10%) · FineMath 0.50B (10%) · Stack-Edu 0.50B code (10%) · Wikipedia 0.50B (10%). All 56 shards sha1-fingerprinted in the project's run manifest.
Stage 2 — context splice 512→1024 + prose continuation, 6.20B tokens ("prose_mix"): FineWeb 35% · Gutenberg 15% · Wikipedia 10% · DCLM 10% · FineWeb-Edu 10% · Cosmopedia 8% · FineMath 6% · Stack-Edu 6%. This is the checkpoint whose bits-per-byte matches token-matched Pythia-160m (paper, Table 1b).
Stage 3 — chat SFT, ~0.43B tokens, 2 epochs: 327k chat conversations (SmolTalk-derived + a deduplicated diverse set + 5,472 in-house self-Q&A pairs + 120 persona seeds), 61k self-corpus document chunks, 10k pretrain-replay documents (forgetting guard), 427 reasoning traces (287 R1-derived CoT + 140 general; system-prompt-gated).
Stage 4 — pinned-persona SFT, ~0.19B tokens: identity trained as a pinned tape record (authority channel) rather than system-prompt tokens; in-house synthetic persona corpus.
Sources are public corpora (FineWeb/FineWeb-Edu ODC-By; DCLM CC-BY-4.0; Cosmopedia
Apache-2.0; Wikipedia CC-BY-SA; Gutenberg public domain; SmolTalk Apache-2.0; Stack-Edu
per-repository licenses; FineMath ODC-By) plus in-house synthetic material (self-Q&A,
persona, general-think; the 287 CoT traces are R1-distilled). No private or user data
in any stage. The shipped eval_slice_prose_val.npy is a 24×1025-token fixture drawn
from the Stage-2 validation split.
What is NOT here, and why
Training orchestration, data pipelines, the §12 memory-organ program (labeled ongoing in the paper), and downstream control/steering machinery. The claims those support are either reported with their own dated work-log provenance (paper, Appendix R) or not yet published. This bundle is scoped to verify what the preprint asserts about these frozen artifacts.
Checksums / provenance
Both checkpoints are weights-only exports (optimizer state stripped) of the exact
training checkpoints named in the paper's Appendix R. Verify with:
sha256sum -c SHA256SUMS