Selective reproducibility release for preprint v1.1 (One Object): two frozen 236M checkpoints, load-only model code, verified repro scripts, hashes

README.md

@@ -0,0 +1,68 @@
+# cl33-opLM — selective reproducibility release (preprint v1.1)
+
+Paper: **One Object: Memory, Navigation, and Reportability in an Operator-Only
+Language Model** — https://t3atlas.dev/cl33/paper/ · Live demo: https://cl33.t3atlas.dev
+Author: Garret Sutherland, MirrorEthic LLC.
+
+This bundle contains what is necessary to **independently test the published claims**
+on frozen artifacts. It is deliberately not the training stack: the paper's §12
+program is ongoing and its machinery is not included. Reproducibility surface ≠
+complete source disclosure.
+
+## Contents
+
+| file | what | sha256 |
+|---|---|---|
+| `cl33_oplm_prose_236m.pt` | prose base, 236.5M, step 189307 (Table 1b checkpoint) | `fe407328…6a1fbf` |
+| `cl33_oplm_chat_236m.pt`  | chat/serving model, step 13996 (the cl33.t3atlas.dev model) | `ddd042a6…559535` |
+| `invert_probe_chat.pt`    | reverse-readout probe (held-out top-1 0.860, card inside) | `91201342…ca79ef` |
+| `model_v2.py` + `model.py` + `so33.py` + `wedge.py` + `t3v3_wedge_memory.py` + `tape_memory.py` | model definition (load-only) | — |
+| `repro_bottleneck.py`     | Claim 1: the mandatory operator bottleneck | — |
+| `repro_reverse_readout.py`| Claim 2: the operator stream is a transcript | — |
+| `SHA256SUMS`              | full hashes | — |
+
+Full hashes in `SHA256SUMS`. Deps: `torch`, `transformers`, `datasets` (Python ≥3.10).
+
+## Claim 1 — the operator bottleneck is mandatory
+
+Zero the emitted operators; the model loses its only path to output.
+
+```
+python repro_bottleneck.py --ckpt cl33_oplm_prose_236m.pt
+python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt
+```
+
+Expected (WikiText-103 test, public data, seq 1024 — measured on this exact bundle):
+
+| ckpt | native PPL | ops-off PPL | ratio |
+|---|---|---|---|
+| prose | ≈61 | ≈6,500 | **≈106×** |
+| chat  | ≈186 | ≈21,000 | **≈112×** |
+
+(The paper's 314× is the chat checkpoint on its in-domain validation mix; the claim
+is the order of magnitude, and it holds off-domain.)
+
+## Claim 2 — the operator stream is a readable transcript
+
+A probe that sees ONLY the emitted operators (no token input) decodes the text:
+
+```
+python repro_reverse_readout.py --text "any sentence you like"
+```
+
+Expected: ~0.86 top-1 on typical English (the probe's held-out card prints on load;
+rare words fail toward semantic neighbors — that is the paper's §7 claim, not a bug).
+
+## What is NOT here, and why
+
+Training orchestration, data pipelines, the §12 memory-organ program (labeled ongoing
+in the paper), and downstream control/steering machinery. The claims those support are
+either reported with their own dated work-log provenance (paper, Appendix R) or not
+yet published. This bundle is scoped to verify what the preprint asserts about these
+frozen artifacts.
+
+## Checksums / provenance
+
+Both checkpoints are weights-only exports (optimizer state stripped) of the exact
+training checkpoints named in the paper's Appendix R. Verify with:
+`sha256sum -c SHA256SUMS`

REPRODUCE.md

@@ -0,0 +1,68 @@
+# cl33-opLM — selective reproducibility release (preprint v1.1)
+
+Paper: **One Object: Memory, Navigation, and Reportability in an Operator-Only
+Language Model** — https://t3atlas.dev/cl33/paper/ · Live demo: https://cl33.t3atlas.dev
+Author: Garret Sutherland, MirrorEthic LLC.
+
+This bundle contains what is necessary to **independently test the published claims**
+on frozen artifacts. It is deliberately not the training stack: the paper's §12
+program is ongoing and its machinery is not included. Reproducibility surface ≠
+complete source disclosure.
+
+## Contents
+
+| file | what | sha256 |
+|---|---|---|
+| `cl33_oplm_prose_236m.pt` | prose base, 236.5M, step 189307 (Table 1b checkpoint) | `fe407328…6a1fbf` |
+| `cl33_oplm_chat_236m.pt`  | chat/serving model, step 13996 (the cl33.t3atlas.dev model) | `ddd042a6…559535` |
+| `invert_probe_chat.pt`    | reverse-readout probe (held-out top-1 0.860, card inside) | `91201342…ca79ef` |
+| `model_v2.py` + `model.py` + `so33.py` + `wedge.py` + `t3v3_wedge_memory.py` + `tape_memory.py` | model definition (load-only) | — |
+| `repro_bottleneck.py`     | Claim 1: the mandatory operator bottleneck | — |
+| `repro_reverse_readout.py`| Claim 2: the operator stream is a transcript | — |
+| `SHA256SUMS`              | full hashes | — |
+
+Full hashes in `SHA256SUMS`. Deps: `torch`, `transformers`, `datasets` (Python ≥3.10).
+
+## Claim 1 — the operator bottleneck is mandatory
+
+Zero the emitted operators; the model loses its only path to output.
+
+```
+python repro_bottleneck.py --ckpt cl33_oplm_prose_236m.pt
+python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt
+```
+
+Expected (WikiText-103 test, public data, seq 1024 — measured on this exact bundle):
+
+| ckpt | native PPL | ops-off PPL | ratio |
+|---|---|---|---|
+| prose | ≈61 | ≈6,500 | **≈106×** |
+| chat  | ≈186 | ≈21,000 | **≈112×** |
+
+(The paper's 314× is the chat checkpoint on its in-domain validation mix; the claim
+is the order of magnitude, and it holds off-domain.)
+
+## Claim 2 — the operator stream is a readable transcript
+
+A probe that sees ONLY the emitted operators (no token input) decodes the text:
+
+```
+python repro_reverse_readout.py --text "any sentence you like"
+```
+
+Expected: ~0.86 top-1 on typical English (the probe's held-out card prints on load;
+rare words fail toward semantic neighbors — that is the paper's §7 claim, not a bug).
+
+## What is NOT here, and why
+
+Training orchestration, data pipelines, the §12 memory-organ program (labeled ongoing
+in the paper), and downstream control/steering machinery. The claims those support are
+either reported with their own dated work-log provenance (paper, Appendix R) or not
+yet published. This bundle is scoped to verify what the preprint asserts about these
+frozen artifacts.
+
+## Checksums / provenance
+
+Both checkpoints are weights-only exports (optimizer state stripped) of the exact
+training checkpoints named in the paper's Appendix R. Verify with:
+`sha256sum -c SHA256SUMS`

SHA256SUMS

@@ -0,0 +1,11 @@
+ddd042a6b8bb1a1e07cdebb75fad135e12955528e872f22b40e7d3fe15559535  release/cl33_oplm_chat_236m.pt
+fe407328903731ecf94e5c7173a754311212a1c24ed2cbb92a1978faea6a1fbf  release/cl33_oplm_prose_236m.pt
+9120134a29141485043107b810ec08a01fb374bfe962421e45d9db88c0ca79ef  release/invert_probe_chat.pt
+df6030348482e9bf0e5ba37e2e62dfada8cf6550a0107a6ce86d68c81019bd8c  model.py
+69992c25815d9c77e11b5e8e7fee644c170387c630c3efe379e700c4d2a7000a  model_v2.py
+fa0a11eb90a78b8b36da5ae9ef2c92f92c99df0e0fe7e76fca9b8036d442bc9e  repro_bottleneck.py
+e93fcaaa8eae49a5bf401c8b722f3290d99f7142b3cab528cd06b02c39e9757f  repro_reverse_readout.py
+ff24dcfa83fb633e71fdeb0093c63b05d6118d574a1efd25fa5744978be0cc2f  so33.py
+fb8d17474cf20ec177a9d6c5810f8d66a79a4a0b7070fa4f111174d91d3b2d29  t3v3_wedge_memory.py
+fa70959ec1657c7747b02ec78652c3ef445f7c72f661b36ea68278c4ab466147  tape_memory.py
+692cd36f59622e37ea83cb7c2957928aa2d49dbf27bb4c362815e47cac96dc33  wedge.py

cl33_oplm_chat_236m.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ddd042a6b8bb1a1e07cdebb75fad135e12955528e872f22b40e7d3fe15559535
+size 946034587

cl33_oplm_prose_236m.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe407328903731ecf94e5c7173a754311212a1c24ed2cbb92a1978faea6a1fbf
+size 945975203

invert_probe_chat.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9120134a29141485043107b810ec08a01fb374bfe962421e45d9db88c0ca79ef
+size 211878933

model.py

@@ -0,0 +1,165 @@
+"""Operator-emitting language model (cl33-opLM v0).
+
+The OPERATOR-ONLY thesis, made structural: context reaches the next-token
+prediction ONLY through emitted operators. A causal transformer "emitter"
+produces, per position, a per-block so(3,3) bivector; those generate rotors; a
+reversible matrix-action scan evolves a multi-block SO(3,3) state; the readout
+sees ONLY that state. No residual bypass from the emitter to the readout — so
+if the operators don't carry the information, PPL suffers (that IS the thesis).
+
+  tokens → embed → causal emitter → h_t
+  h_t → op_head → coefs_t (n_blocks, 15)   [clipped, small init]
+  R_t = matrix_exp(Σ coefs_t·G)            [per block, fp32]
+  s_t = R_t · s_{t-1}                       [reversible scan; s_0 learned]
+  logits = readout(LayerNorm(flatten s_t))  [readout sees ONLY the state]
+
+Scan runs in float32 (bf16 diverges — proven on the lattice). Emitter may be
+autocast-bf16; the state path is force-fp32.
+"""
+from __future__ import annotations
+
+import math
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from so33 import build_generators, rotor_from_coefs, N_GEN  # type: ignore
+from wedge import GradeTower  # type: ignore
+
+# per-block readout feature dims: g1 + g2 + g3 + g4 + g5 + g6
+_GRADE1, _GRADE2, _GRADE3, _GRADE4, _GRADE5, _GRADE6 = 6, 15, 20, 15, 6, 1
+_TOWER_DIM = _GRADE1 + _GRADE2 + _GRADE3 + _GRADE4 + _GRADE5 + _GRADE6  # 63
+
+
+@dataclass
+class OpEmitConfig:
+    vocab_size: int = 8192
+    d_model: int = 384
+    n_layers: int = 6
+    n_heads: int = 6
+    d_ff: int = 1536
+    max_seq_len: int = 256
+    n_blocks: int = 16          # multi-block SO(3,3) state (6·n_blocks dims)
+    coef_clip: float = 1.0      # per-block bivector L2-norm cap (bounds growth)
+    op_init_scale: float = 0.02 # rotors start ≈ identity
+    use_grade_tower: bool = True  # readout sees full grade tower per block
+    dropout: float = 0.0
+
+
+class EmitterBlock(nn.Module):
+    def __init__(self, c: OpEmitConfig):
+        super().__init__()
+        self.n_heads = c.n_heads
+        self.d_head = c.d_model // c.n_heads
+        self.qkv = nn.Linear(c.d_model, 3 * c.d_model, bias=False)
+        self.o = nn.Linear(c.d_model, c.d_model, bias=False)
+        self.norm1 = nn.LayerNorm(c.d_model)
+        self.norm2 = nn.LayerNorm(c.d_model)
+        self.mlp = nn.Sequential(nn.Linear(c.d_model, c.d_ff), nn.GELU(),
+                                 nn.Linear(c.d_ff, c.d_model))
+
+    def forward(self, x):
+        B, T, D = x.shape
+        h = self.norm1(x)
+        qkv = self.qkv(h).reshape(B, T, 3, self.n_heads, self.d_head)
+        q, k, v = qkv.unbind(2)
+        q, k, v = (t.transpose(1, 2) for t in (q, k, v))       # (B,H,T,dh)
+        a = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        a = a.transpose(1, 2).reshape(B, T, D)
+        x = x + self.o(a)
+        x = x + self.mlp(self.norm2(x))
+        return x
+
+
+class OpEmitLM(nn.Module):
+    def __init__(self, c: OpEmitConfig):
+        super().__init__()
+        self.c = c
+        self.tok_embed = nn.Embedding(c.vocab_size, c.d_model)
+        self.pos_embed = nn.Embedding(c.max_seq_len, c.d_model)
+        self.blocks = nn.ModuleList([EmitterBlock(c) for _ in range(c.n_layers)])
+        self.norm = nn.LayerNorm(c.d_model)
+        self.op_head = nn.Linear(c.d_model, c.n_blocks * N_GEN)
+        nn.init.normal_(self.op_head.weight, std=1e-3)
+        nn.init.zeros_(self.op_head.bias)
+        # learned initial state per block (nonzero so rotors have something to act on)
+        self.s0 = nn.Parameter(torch.randn(c.n_blocks, 6) * 0.5)
+        feat_per_block = _TOWER_DIM if c.use_grade_tower else 6
+        self.feat_dim = c.n_blocks * feat_per_block
+        self.tower = GradeTower() if c.use_grade_tower else None
+        self.state_norm = nn.LayerNorm(self.feat_dim)
+        self.readout = nn.Linear(self.feat_dim, c.vocab_size, bias=False)
+        self.register_buffer("G", build_generators(dtype=torch.float32), persistent=False)
+
+    def emit(self, idx):
+        B, T = idx.shape
+        pos = torch.arange(T, device=idx.device)
+        x = self.tok_embed(idx) + self.pos_embed(pos)[None]
+        for blk in self.blocks:
+            x = blk(x)
+        h = self.norm(x)
+        coefs = self.op_head(h).view(B, T, self.c.n_blocks, N_GEN)
+        coefs = coefs * self.c.op_init_scale
+        # per-block L2-norm clip → bounded rotors → bounded state growth
+        n = coefs.norm(dim=-1, keepdim=True)
+        coefs = coefs * (self.c.coef_clip / n.clamp_min(self.c.coef_clip))
+        return coefs                                            # (B,T,nb,15)
+
+    def scan(self, coefs, ablate=False):
+        """Reversible matrix-action scan. Returns S (B,T,nb,6). fp32 forced."""
+        B, T, nb, _ = coefs.shape
+        coefs = coefs.float()
+        if ablate:
+            R = torch.eye(6, device=coefs.device).expand(B, T, nb, 6, 6)
+        else:
+            R = rotor_from_coefs(coefs, self.G.float())         # (B,T,nb,6,6)
+        s = self.s0.float().expand(B, nb, 6).contiguous()
+        s = s / s.norm(dim=-1, keepdim=True).clamp_min(1e-6)
+        S = torch.empty(B, T, nb, 6, device=coefs.device, dtype=torch.float32)
+        for t in range(T):
+            s = torch.einsum("bnij,bnj->bni", R[:, t], s)
+            # per-block unit-norm projection: bounds growth (boosts amplify ‖s‖),
+            # state lives on (S⁵)^nb. Matches the lattice's normalized recurrence.
+            s = s / s.norm(dim=-1, keepdim=True).clamp_min(1e-6)
+            S[:, t] = s
+        return S
+
+    def forward(self, idx, targets=None, ablate_scan=False):
+        """ablate_scan=True → R→IDENTITY (state frozen at s0) but coefs/tower
+        stay LIVE. This is the BYPASS TEST: if the model still predicts well
+        with the rotors off, the grade tower is feeding operators to the readout
+        past the state recurrence (a bypass) — the causal claim is dead. Correct
+        pass: ablated PPL ≈ unigram floor. (Reported as absolute PPL, not a
+        ratio — the ratio inflates mechanically as trained PPL drops.)"""
+        B, T = idx.shape
+        coefs = self.emit(idx)                                 # (B,T,nb,15)
+        S = self.scan(coefs, ablate=ablate_scan)               # (B,T,nb,6) grade-1
+        if self.tower is not None:
+            # causal shifts of the operator sequence (grade-2)
+            B_prev = torch.zeros_like(coefs)
+            B_prev[:, 1:] = coefs[:, :-1]
+            B_prev2 = torch.zeros_like(coefs)
+            B_prev2[:, 2:] = coefs[:, :-2]
+            g3, g4, g5, g6 = self.tower(S, coefs, B_prev, B_prev2)
+            feat = torch.cat([S, coefs, g3, g4, g5, g6], dim=-1)  # (B,T,nb,63)
+            feat = feat.reshape(B, T, -1)
+        else:
+            feat = S.reshape(B, T, -1)
+        feat = self.state_norm(feat)
+        logits = self.readout(feat)                            # (B,T,vocab)
+        loss = None
+        if targets is not None:
+            loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                                   targets.reshape(-1))
+        return logits, loss
+
+    def param_count(self):
+        return sum(p.numel() for p in self.parameters() if p.requires_grad)
+
+
+__all__ = ["OpEmitConfig", "OpEmitLM"]

model_v2.py

@@ -0,0 +1,332 @@
+"""cl33-opLM v2 — operator-native attention over the state trajectory.
+
+The v0/widen finding: the operator-only bottleneck forces a RECURRENT state that
+compresses history; widening independent blocks doesn't beat compression. The
+fix (Garret): restore attention, but map it 1:1 onto the algebra so it stays
+operator-derived (causal transparency preserved).
+
+Operator-native attention (per block = per head):
+  state scan:  s_t = R(B_state_t) · s_{t-1}          (as before, reversible)
+  query/key:   Q_t = R(B_q_t) · s_t,  K_s = R(B_k_s) · s_s   (emitted rotors)
+  score:       ⟨Q_t, K_s⟩_η   (η-metric inner product, causal)
+  attend:      a_t = Σ_s softmax(score)_ts · s_s      (values = the states)
+  readout:     LayerNorm(flatten a_t) → vocab
+
+Because rotors preserve η, score(t,s) = ⟨s_t, (R_q⁻¹R_k)·s_s⟩_η — attention is
+the alignment of the current state with a LEARNED-RELATIVELY-ROTATED past state.
+Everything is operator-derived → operator-only bottleneck holds → the R_state→
+identity bypass test still collapses to unigram (all states=s0 ⇒ attention over
+identical values ⇒ constant readout).
+
+Scan + attention in fp32 (bf16 proven-unsafe on the recurrence).
+"""
+from __future__ import annotations
+
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from so33 import build_generators, rotor_from_coefs, N_GEN, ETA  # type: ignore
+from model import EmitterBlock  # reuse the emitter block
+from wedge import GradeTower    # grade tower for short-context (current operator)
+from t3v3_wedge_memory import WedgeMemory, TokenCopyMemory  # associative / copy memory
+from tape_memory import TapeMemory  # reversible-tape read (address by token, flow algebra)
+
+
+@dataclass
+class OpEmitV2Config:
+    vocab_size: int = 8192
+    d_model: int = 384
+    n_layers: int = 6
+    n_heads: int = 6
+    d_ff: int = 1536
+    max_seq_len: int = 256
+    n_blocks: int = 32          # = attention heads (each attends in its SO(3,3))
+    coef_clip: float = 1.0
+    op_init_scale: float = 0.02
+    qk_init_scale: float = 0.1  # q/k rotors can be larger (learn what to attend)
+    use_grade_tower: bool = True  # v2.2: current-operator grade tower in readout
+    use_wedge_memory: bool = False  # wedge bivector associative memory (KV binding)
+    wedge_key_source: str = "operator"  # "operator" | "state" | "token"
+    wedge_value_source: str = "same"    # "same" | "operator" (G1: algebra value w/ token key)
+    wedge_delta_rule: bool = False      # DeltaNet residual write (D3 interference fix)
+    use_token_copy: bool = False    # token-content copy channel (measured transparency cost)
+    use_tape_memory: bool = False   # reversible-tape read (REVERSIBLE_TAPE_DESIGN.md)
+    tape_value_mode: str = "increment"  # "displacement" | "state" | "increment"
+    tape_dual_address: bool = False  # v2.5: + token-faithful exact-match channel (TAPE_ADDRESSING_V25.md)
+    tape_addr: str = "token"         # v2.8: "token" (address by token embedding) | "operator" (address
+    #                                  by the emitted LM operator — native relational key, not the
+    #                                  redundant token channel CE routes around). Pairs with tape_compose.
+    tape_compose: bool = False       # v2.8: gp-COMPOSE recalled operator with the query rotor before
+    #                                  readout (genesis marriage compose + v2.7 product-with-query). The
+    #                                  recalled op stops being read in isolation — it composes with the
+    #                                  current computation. Needs tape_value_mode=increment (15-d recall).
+    tape_compose_mode: str = "add"   # "add" = additive zero-init logit (SAFE, out of LN; augments the
+    #                                  distribution). "tilt" = recalled operator (gated, zero-init) TILTS
+    #                                  the decoded state itself, decoded by the normal readout (augments
+    #                                  BEHAVIOR; in the LN → tests whether it survives tape-silencing).
+    scan_only: bool = False         # zero the O(T²) attention — tape/scan carry history
+    dropout: float = 0.0
+    grad_checkpoint: bool = False   # activation-checkpoint the emitter transformer blocks: recompute
+    #                                 in backward instead of storing (bit-identical forward, unchanged
+    #                                 architecture/reversibility). Frees the dominant activation memory
+    #                                 → larger batch → better sequential-scan SM occupancy on small GPUs.
+
+
+class OpEmitLMv2(nn.Module):
+    def __init__(self, c: OpEmitV2Config):
+        super().__init__()
+        self.c = c
+        self.tok_embed = nn.Embedding(c.vocab_size, c.d_model)
+        self.pos_embed = nn.Embedding(c.max_seq_len, c.d_model)
+        self.blocks = nn.ModuleList([EmitterBlock(c) for _ in range(c.n_layers)])
+        self.norm = nn.LayerNorm(c.d_model)
+        # emit 3 bivectors per block: state-evolution, query-rotor, key-rotor
+        self.op_head = nn.Linear(c.d_model, c.n_blocks * 3 * N_GEN)
+        nn.init.normal_(self.op_head.weight, std=1e-3)
+        nn.init.zeros_(self.op_head.bias)
+        self.s0 = nn.Parameter(torch.randn(c.n_blocks, 6) * 0.5)
+        # readout features per block:
+        #   attended(6) + s_t(6=g1)  [residual: long-context attn + current state]
+        #   + grade tower g2(15)+g3(20)+g4(15)+g5(6)+g6(1)=57  [v2.2: current
+        #     operator, rich at pos 1 — fixes the rotation-of-s0 short-context
+        #     limit]. All operator-derived → bottleneck preserved.
+        self.tower = GradeTower() if c.use_grade_tower else None
+        self.wedge = (WedgeMemory(key_source=getattr(c, "wedge_key_source", "operator"),
+                                  value_source=getattr(c, "wedge_value_source", "same"),
+                                  delta_rule=getattr(c, "wedge_delta_rule", False),
+                                  d_model=c.d_model, nb=c.n_blocks)
+                      if getattr(c, "use_wedge_memory", False) else None)
+        self.token_copy = (TokenCopyMemory(c.d_model, c.n_blocks)
+                           if getattr(c, "use_token_copy", False) else None)
+        self.tape = (TapeMemory(c.d_model, c.n_blocks,
+                                value_mode=getattr(c, "tape_value_mode", "increment"),
+                                dual_address=getattr(c, "tape_dual_address", False),
+                                addr=getattr(c, "tape_addr", "token"))
+                     if getattr(c, "use_tape_memory", False) else None)
+        # readout features per block: +6 each for the wedge read and the copy read.
+        # The TAPE is deliberately NOT in this vector — it is a SEPARATE post-LayerNorm additive
+        # readout term (see assemble). Reason: concatenating the tape into the shared LayerNorm lets
+        # any tape contribution perturb the normalization of the base features, so the optimizer
+        # silences the tape to protect the base LM (observed: v2.4 gate 0.01→0.0005). Keeping the
+        # tape OUT of the LN leaves the base bit-exact (clean warm-start) and lets the tape co-train.
+        feat_per_block = (12 + (6 if self.wedge is not None else 0)
+                          + (6 if self.token_copy is not None else 0)
+                          + (57 if c.use_grade_tower else 0))
+        self.state_norm = nn.LayerNorm(c.n_blocks * feat_per_block)
+        self.readout = nn.Linear(c.n_blocks * feat_per_block, c.vocab_size, bias=False)
+        # tape = additive side channel, ZERO-INIT readout → off at step 0, learns on. The zero-init
+        # readout IS the clean off-switch (no gate scalar needed, no LN-suppression dynamic).
+        if self.tape is not None:
+            self.tape_readout = nn.Linear(self.tape.out_dim() * c.n_blocks, c.vocab_size, bias=False)
+            nn.init.zeros_(self.tape_readout.weight)
+        # v2.8 compose channel: recalled operator gp-composed with the query rotor, applied to state.
+        # Separate ZERO-INIT readout (same off-switch pattern) → step-0 ≡ base, co-trains on.
+        self.tape_compose = getattr(c, "tape_compose", False) and self.tape is not None
+        if self.tape_compose:
+            assert self.tape.out_dim() == 15, "tape_compose needs 15-d bivector recall (tape_value_mode=increment)"
+            if getattr(c, "tape_compose_mode", "add") == "tilt":
+                # gated recalled operator tilts the decoded state; per-block gate ZERO-INIT → R_tilt=I,
+                # step-0 state unchanged. In the LN via parts=[a, S_tilt] → the behavior-augmenting arm.
+                self.tape_tilt_gate = nn.Parameter(torch.zeros(c.n_blocks, 1))
+            else:
+                self.tape_compose_readout = nn.Linear(c.n_blocks * 6, c.vocab_size, bias=False)
+                nn.init.zeros_(self.tape_compose_readout.weight)
+        self.register_buffer("G", build_generators(dtype=torch.float32), persistent=False)
+        self.register_buffer("eta", ETA.clone(), persistent=False)
+
+    def emit(self, idx):
+        B, T = idx.shape
+        pos = torch.arange(T, device=idx.device)
+        x = self.tok_embed(idx) + self.pos_embed(pos)[None]
+        if getattr(self.c, "grad_checkpoint", False) and self.training:
+            from torch.utils.checkpoint import checkpoint
+            for blk in self.blocks:
+                x = checkpoint(blk, x, use_reentrant=False)   # recompute in backward, free activations
+        else:
+            for blk in self.blocks:
+                x = blk(x)
+        h = self.norm(x)
+        raw = self.op_head(h).view(B, T, self.c.n_blocks, 3, N_GEN)
+        Bs, Bq, Bk = raw.unbind(-2)                            # each (B,T,nb,15)
+
+        def clip(bv, scale):
+            bv = bv * scale
+            n = bv.norm(dim=-1, keepdim=True)
+            return bv * (self.c.coef_clip / n.clamp_min(self.c.coef_clip))
+        return (clip(Bs, self.c.op_init_scale),
+                clip(Bq, self.c.qk_init_scale),
+                clip(Bk, self.c.qk_init_scale))
+
+    @torch.no_grad()
+    def emitter_hidden(self, idx):
+        """The emitter's final hidden h_t (baseline for recoverability probes)."""
+        B, T = idx.shape
+        pos = torch.arange(T, device=idx.device)
+        x = self.tok_embed(idx) + self.pos_embed(pos)[None]
+        for blk in self.blocks:
+            x = blk(x)
+        return self.norm(x)                                    # (B,T,d_model)
+
+    def scan(self, Bs, ablate=False):
+        """Reversible state recurrence. Returns S (B,T,nb,6), fp32."""
+        B, T, nb, _ = Bs.shape
+        Bs = Bs.float()
+        R = (torch.eye(6, device=Bs.device).expand(B, T, nb, 6, 6) if ablate
+             else rotor_from_coefs(Bs, self.G.float()))
+        s = self.s0.float().expand(B, nb, 6).contiguous()
+        s = s / s.norm(dim=-1, keepdim=True).clamp_min(1e-6)
+        S = torch.empty(B, T, nb, 6, device=Bs.device, dtype=torch.float32)
+        for t in range(T):
+            s = torch.einsum("bnij,bnj->bni", R[:, t], s)
+            s = s / s.norm(dim=-1, keepdim=True).clamp_min(1e-6)
+            S[:, t] = s
+        return S
+
+    def scan_parallel(self, Bs, ablate=False):
+        """Parallel associative scan (PARALLEL_SCAN_SPEC): the per-step normalize cancels, so
+        S_t = normalize(P_t·s0) with P_t = R_t···R_0 a prefix product. Hillis-Steele scan under the
+        associative operator A∘B = normalize_F(A·B) (Frobenius-normalized to avoid boost overflow).
+        O(log T) depth. GATED against scan() — must match bit-for-bit before it replaces it."""
+        B, T, nb, _ = Bs.shape
+        Bs = Bs.float()
+        R = (torch.eye(6, device=Bs.device).expand(B, T, nb, 6, 6).contiguous() if ablate
+             else rotor_from_coefs(Bs, self.G.float()))
+        def nF(M):
+            return M / (M.reshape(*M.shape[:-2], 36).norm(dim=-1)[..., None, None] + 1e-30)
+        P = nF(R)                                                   # (B,T,nb,6,6)
+        I = torch.eye(6, device=Bs.device, dtype=P.dtype).expand(B, 1, nb, 6, 6)
+        idx = torch.arange(T, device=Bs.device)
+        d = 1
+        while d < T:
+            right = torch.cat([I.expand(B, d, nb, 6, 6), P[:, :T - d]], dim=1)   # right[t]=P[t-d], I for t<d
+            combined = nF(P @ right)                                # newest(left) @ older(right), correct order
+            mask = (idx >= d)[None, :, None, None, None]
+            P = torch.where(mask, combined, P)
+            d *= 2
+        s0 = self.s0.float().expand(B, nb, 6)
+        s0 = s0 / s0.norm(dim=-1, keepdim=True).clamp_min(1e-6)
+        S = torch.einsum("btnij,bnj->btni", P, s0)
+        return S / S.norm(dim=-1, keepdim=True).clamp_min(1e-6)
+
+    def attend(self, S, Bq, Bk):
+        """Operator-native causal attention over the state trajectory.
+        S (B,T,nb,6); Bq,Bk (B,T,nb,15). Returns attended (B,T,nb,6)."""
+        B, T, nb, _ = S.shape
+        Rq = rotor_from_coefs(Bq.float(), self.G.float())      # (B,T,nb,6,6)
+        Rk = rotor_from_coefs(Bk.float(), self.G.float())
+        Q = torch.einsum("btnij,btnj->btni", Rq, S)            # (B,T,nb,6)
+        K = torch.einsum("btnij,btnj->btni", Rk, S)
+        # η-metric scores: ⟨Q_t, K_s⟩_η, per block/head. (B,nb,T,T)
+        Kw = K * self.eta.to(K.dtype)                          # apply η to keys
+        scores = torch.einsum("btni,bsni->bnts", Q, Kw) / (6 ** 0.5)
+        causal = torch.triu(torch.ones(T, T, device=S.device, dtype=torch.bool), 1)
+        scores = scores.masked_fill(causal, float("-inf"))
+        A = F.softmax(scores, dim=-1)                          # (B,nb,T,T)
+        attended = torch.einsum("bnts,bsni->btni", A, S)       # values = states
+        return attended
+
+    def assemble(self, Bs, Bq, Bk, scan_only=False, tok_emb=None):
+        """Full pass from emitted operators → logits (scan + attention + tower
+        + readout). Separated from emit() so CONTROL interventions can perturb
+        the operators and re-run only the downstream. Returns (B,T,vocab).
+
+        scan_only=True zeroes the cross-position attention path, so recall must
+        come from the recurrent reversible state alone (the fair vs-xLSTM memory
+        test — isolates the linear-recurrent memory from the O(T²) attention)."""
+        B, T = Bs.shape[:2]
+        S = self.scan(Bs, ablate=False)                        # (B,T,nb,6)
+        # tape read (computed here so a 'tilt' compose can act on the decoded state below)
+        tape_read = None
+        if self.tape is not None and tok_emb is not None:
+            R_state = (rotor_from_coefs(Bs.float(), self.G.float())
+                       if (self.tape.value_mode in ("displacement", "multi")
+                           or getattr(self.tape, "addr", "token") == "target") else None)
+            tape_read = self.tape(S, Bs, R_state, tok_emb, self.eta, Bq=Bq)   # (B,T,nb,out_dim)
+        # v2.8 tilt-compose: the recalled operator (gated, zero-init) tilts the state that gets decoded
+        # by the NORMAL readout — augments behavior, then decoded like usual.
+        S_ro = S
+        if (tape_read is not None and getattr(self, "tape_compose", False)
+                and getattr(self.c, "tape_compose_mode", "add") == "tilt"):
+            tilt_biv = self.tape_tilt_gate * tape_read.float()        # (B,T,nb,15); gate 0 → identity
+            R_tilt = rotor_from_coefs(tilt_biv, self.G.float())       # (B,T,nb,6,6)
+            S_ro = torch.einsum("btnij,btnj->btni", R_tilt, S)        # recalled op tilts the decoded state
+        a = torch.zeros_like(S) if scan_only else self.attend(S, Bq, Bk)
+        parts = [a, S_ro]
+        if self.wedge is not None:
+            # O(T) linear-recurrent associative memory. operator mode keys on the
+            # per-token emitted operators (content-bearing); state mode (ablation)
+            # keys on the transported state + adjoint-transports the memory.
+            Rq = Rk = R_state = None
+            if self.wedge.key_source == "state":
+                G32 = self.G.float()
+                Rq = rotor_from_coefs(Bq.float(), G32)
+                Rk = rotor_from_coefs(Bk.float(), G32)
+                R_state = rotor_from_coefs(Bs.float(), G32)
+            parts.append(self.wedge(S, Rq, Rk, self.eta, R_state=R_state,
+                                    Bs=Bs, Bq=Bq, Bk=Bk, tok_emb=tok_emb))  # r_t (B,T,nb,6)
+        if self.token_copy is not None and tok_emb is not None:
+            # token-content copy channel (bypasses operators; transparency cost measured)
+            parts.append(self.token_copy(tok_emb))             # (B,T,nb,6)
+        # (tape_read computed above, before the state-tilt; consumed as a SEPARATE post-LN additive
+        # term below in "add" mode, or already applied as a state tilt above in "tilt" mode.)
+        if self.tower is not None:
+            Bs_prev = torch.zeros_like(Bs); Bs_prev[:, 1:] = Bs[:, :-1]
+            Bs_prev2 = torch.zeros_like(Bs); Bs_prev2[:, 2:] = Bs[:, :-2]
+            g3, g4, g5, g6 = self.tower(S, Bs, Bs_prev, Bs_prev2)
+            parts += [Bs, g3, g4, g5, g6]
+        feat = self.state_norm(torch.cat(parts, dim=-1).reshape(B, T, -1))
+        logits = self.readout(feat)
+        if tape_read is not None:                                     # additive, post-LN, zero-init
+            logits = logits + self.tape_readout(tape_read.reshape(B, T, -1).to(logits.dtype))
+        if (tape_read is not None and getattr(self, "tape_compose", False)
+                and getattr(self.c, "tape_compose_mode", "add") == "add"):
+            # gp-COMPOSE the recalled operator with the current query rotor (exact matrix/rotor
+            # composition — lossless in the grade-1 action), apply to the state. The recalled
+            # operation now composes with the current computation instead of being read in isolation:
+            # genesis-marriage compose + v2.7's product-with-query. Zero-init readout → off at step 0.
+            G32 = self.G.float()
+            R_rec = rotor_from_coefs(tape_read.float(), G32)           # (B,T,nb,6,6) recalled operator
+            R_q = rotor_from_coefs(Bq.float(), G32)                    # (B,T,nb,6,6) current query
+            R_comp = torch.einsum("btnij,btnjk->btnik", R_rec, R_q)    # exact rotor composition
+            comp = torch.einsum("btnij,btnj->btni", R_comp, S)         # composed op applied to state
+            logits = logits + self.tape_compose_readout(comp.reshape(B, T, -1).to(logits.dtype))
+        return logits
+
+    def forward(self, idx, targets=None, ablate_scan=False, amp_emit=False):
+        # amp_emit: run the transformer emitter in fp16 (tensor-core speedup, safe)
+        # but the operator algebra (scan/attend/tower/readout) in fp32 — fp16 there
+        # overflows (η-metric scores + nested wedge products) → NaN. Split keeps the
+        # bulk of compute fast while the delicate algebra stays numerically exact.
+        if amp_emit:
+            with torch.autocast("cuda", dtype=torch.float16):
+                Bs, Bq, Bk = self.emit(idx)
+            Bs, Bq, Bk = Bs.float(), Bq.float(), Bk.float()
+        else:
+            Bs, Bq, Bk = self.emit(idx)
+        if ablate_scan:
+            Bs = torch.zeros_like(Bs)                          # transparency test
+        # tok_emb (raw embedding) feeds the token-copy / token-key channels; on the
+        # ablate path it is retained → bypass ratio then reflects how much those
+        # channels carry prediction around the (zeroed) operators.
+        tok_emb = self.tok_embed(idx) if (self.token_copy is not None
+                  or self.tape is not None
+                  or (self.wedge is not None and self.wedge.key_source == "token")) else None
+        logits = self.assemble(Bs, Bq, Bk, scan_only=getattr(self.c, "scan_only", False),
+                               tok_emb=tok_emb)
+        loss = None
+        if targets is not None:
+            loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]),
+                                   targets.reshape(-1))
+        return logits, loss
+
+    def param_count(self):
+        return sum(p.numel() for p in self.parameters() if p.requires_grad)
+
+
+__all__ = ["OpEmitV2Config", "OpEmitLMv2"]

repro_bottleneck.py

@@ -0,0 +1,59 @@
+"""Reproduce the mandatory-operator-bottleneck claim on a released cl33-opLM checkpoint.
+
+Zeroing the emitted operators (Bs=0 everywhere: identity rotors in the scan, zeroed
+readout features) multiplies perplexity by orders of magnitude — the model has no
+other path to output. Paper: "One Object" §1/§2/§7; 314x measured on the chat
+checkpoint (prose validation); expect the same order of magnitude on WikiText-103.
+
+Usage:  python repro_bottleneck.py --ckpt cl33_oplm_chat_236m.pt [--iters 20]
+Deps:   torch, transformers, datasets  (model_v2.py + so33.py from this bundle)
+"""
+import argparse, math, sys
+from pathlib import Path
+import torch, torch.nn.functional as F
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from model_v2 import OpEmitV2Config, OpEmitLMv2
+
+ap = argparse.ArgumentParser()
+ap.add_argument("--ckpt", default="cl33_oplm_chat_236m.pt")
+ap.add_argument("--iters", type=int, default=20)
+ap.add_argument("--seq", type=int, default=1024)
+a = ap.parse_args()
+dev = "cuda" if torch.cuda.is_available() else "cpu"
+
+d = torch.load(a.ckpt, map_location="cpu", weights_only=False)
+cfg = OpEmitV2Config(**{k: v for k, v in d["config"].items()
+                        if k in OpEmitV2Config.__dataclass_fields__})
+m = OpEmitLMv2(cfg); m.load_state_dict(d["model"]); m.eval().to(dev)
+for p in m.parameters(): p.requires_grad_(False)
+print(f"loaded {a.ckpt} | {sum(p.numel() for p in m.parameters())/1e6:.1f}M params | step {d.get('step')}")
+
+from transformers import GPT2TokenizerFast
+from datasets import load_dataset
+tok = GPT2TokenizerFast.from_pretrained("gpt2")
+text = "\n\n".join(load_dataset("Salesforce/wikitext", "wikitext-103-raw-v1",
+                                split="test")["text"])
+ids = tok(text, add_special_tokens=False)["input_ids"]
+print(f"wikitext-103 test: {len(ids)/1e6:.2f}M tokens")
+
+def ce_pass(zero_ops: bool):
+    tot, n = 0.0, 0
+    for i in range(a.iters):
+        s = i * a.seq
+        x = torch.tensor([ids[s:s+a.seq]], device=dev)
+        y = torch.tensor([ids[s+1:s+a.seq+1]], device=dev)
+        with torch.no_grad():
+            Bs, Bq, Bk = m.emit(x)
+            if zero_ops: Bs = torch.zeros_like(Bs)
+            logits = m.assemble(Bs, Bq, Bk, scan_only=getattr(cfg, "scan_only", False))
+            if isinstance(logits, tuple): logits = logits[0]
+            tot += float(F.cross_entropy(logits.reshape(-1, logits.shape[-1]).float(),
+                                         y.reshape(-1), reduction="sum"))
+            n += y.numel()
+    return tot / n
+
+ce_nat = ce_pass(False); ce_off = ce_pass(True)
+print(f"\n  native : CE {ce_nat:.3f}  PPL {math.exp(ce_nat):9.1f}")
+print(f"  ops off: CE {ce_off:.3f}  PPL {math.exp(ce_off):9.1f}")
+print(f"  BOTTLENECK RATIO: {math.exp(ce_off)/math.exp(ce_nat):.0f}x  "
+      f"(paper: 314x on chat ckpt / prose val; order of magnitude is the claim)")

repro_reverse_readout.py

@@ -0,0 +1,47 @@
+"""Reproduce the operator-transcript claim: decode text back out of the operator
+stream alone. Paper §7 (inversion battery, 0.86 top-1/50k on scale-C; the released
+probe is trained on the chat checkpoint's operators — held-out top-1 0.860).
+
+Usage:  python repro_reverse_readout.py [--text "any text you like"]
+"""
+import argparse, sys
+from pathlib import Path
+import torch
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from model_v2 import OpEmitV2Config, OpEmitLMv2
+
+ap = argparse.ArgumentParser()
+ap.add_argument("--ckpt", default="cl33_oplm_chat_236m.pt")
+ap.add_argument("--probe", default="invert_probe_chat.pt")
+ap.add_argument("--text", default="The reverse readout decodes the conversation "
+                                  "from the operator record alone, with no token input.")
+a = ap.parse_args()
+dev = "cuda" if torch.cuda.is_available() else "cpu"
+
+d = torch.load(a.ckpt, map_location="cpu", weights_only=False)
+cfg = OpEmitV2Config(**{k: v for k, v in d["config"].items()
+                        if k in OpEmitV2Config.__dataclass_fields__})
+m = OpEmitLMv2(cfg); m.load_state_dict(d["model"]); m.eval().to(dev)
+import torch.nn as nn
+pd = torch.load(a.probe, map_location="cpu", weights_only=False)
+probe = nn.Sequential(nn.Linear(pd["in_dim"], pd["hidden"]), nn.GELU(),
+                      nn.Linear(pd["hidden"], pd["vocab"]))
+probe.load_state_dict(pd["probe"]); probe.eval().to(dev)
+# stored held-out results ride with the artifact:
+print("probe card:", {k: round(v, 3) for k, v in pd["results"].items()
+                      if isinstance(v, float)})
+
+from transformers import GPT2TokenizerFast
+tok = GPT2TokenizerFast.from_pretrained("gpt2")
+ids = tok(a.text, add_special_tokens=False)["input_ids"]
+x = torch.tensor([ids], device=dev)
+with torch.no_grad():
+    Bs, Bq, Bk = m.emit(x)
+    t = len(ids)
+    feats = torch.cat([Bs.reshape(1, t, -1), Bq.reshape(1, t, -1),
+                       Bk.reshape(1, t, -1)], -1).float()   # [Bs_480|Bq_480|Bk_480]
+    pred = probe(feats).argmax(-1)[0]
+hits = sum(int(p == t) for p, t in zip(pred.tolist(), ids))
+print(f"tokens: {len(ids)} | decoded from operators alone, top-1 exact: "
+      f"{hits}/{len(ids)} = {hits/len(ids):.2f}")
+print("decoded:", tok.decode(pred.tolist()))

so33.py

@@ -0,0 +1,66 @@
+"""so(3,3) generator basis + matrix-exp rotor + reversible matrix-action scan.
+
+The operator-emitting LM works in a multi-block SO(3,3) state: each block is a
+6-d vector; an emitted 15-coef bivector generates a per-block rotor R = exp(Ω),
+Ω ∈ so(3,3), and the state evolves by matrix action s_t = R_t · s_{t-1}.
+
+Signature η = diag(-1,-1,-1,+1,+1,+1) (matches cl33_t3lm/kge_inference.py).
+so(3,3) = { X : XᵀηX preserved } = { X : ηX antisymmetric }. A basis is
+G_ij = η (e_i e_jᵀ - e_j e_iᵀ) over the 15 pairs i<j — 6 compact rotations
+(both axes same-sign) + 9 boosts (mixed-sign).
+
+Reversibility (the interpretability handle): every rotor preserves η, so
+R⁻¹ = η Rᵀ η exactly — no inverse needed. The scan is bit-reversible.
+
+Everything here runs in float32 (bf16 diverges — proven on the lattice).
+"""
+from __future__ import annotations
+
+import torch
+
+ETA = torch.tensor([-1., -1., -1., 1., 1., 1.])          # (6,)
+PAIRS = [(i, j) for i in range(6) for j in range(i + 1, 6)]  # 15 pairs
+N_GEN = len(PAIRS)                                        # 15
+
+
+def build_generators(device=None, dtype=torch.float32) -> torch.Tensor:
+    """(15, 6, 6) so(3,3) generator matrices G_ij = η(e_i e_jᵀ - e_j e_iᵀ)."""
+    eta = ETA.to(device=device, dtype=dtype)
+    G = torch.zeros(N_GEN, 6, 6, device=device, dtype=dtype)
+    for k, (i, j) in enumerate(PAIRS):
+        A = torch.zeros(6, 6, device=device, dtype=dtype)
+        A[i, j] = 1.0
+        A[j, i] = -1.0
+        G[k] = eta[:, None] * A                          # η @ A (η diagonal)
+    return G
+
+
+def rotor_from_coefs(coefs: torch.Tensor, G: torch.Tensor) -> torch.Tensor:
+    """coefs (..., 15) -> R (..., 6, 6) = matrix_exp(Σ coefs·G).
+
+    Runs in the coefs' dtype; caller must keep this in float32.
+    """
+    Omega = torch.einsum("...k,kij->...ij", coefs, G)    # (...,6,6) in so(3,3)
+    return torch.linalg.matrix_exp(Omega)
+
+
+def rotor_inverse(R: torch.Tensor) -> torch.Tensor:
+    """R⁻¹ = η Rᵀ η — exact, no solve. R (...,6,6)."""
+    eta = ETA.to(device=R.device, dtype=R.dtype)
+    Rt = R.transpose(-1, -2)
+    return eta[..., :, None] * Rt * eta[..., None, :]
+
+
+def eta_inner(a: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
+    """η-metric inner product ⟨a,b⟩_η over the last dim (6). a,b (...,6)."""
+    eta = ETA.to(device=a.device, dtype=a.dtype)
+    return (a * eta * b).sum(-1)
+
+
+def q_invariant(v: torch.Tensor) -> torch.Tensor:
+    """Q(v) = -v0²-v1²-v2²+v3²+v4²+v5²  (preserved by every rotor)."""
+    return eta_inner(v, v)
+
+
+__all__ = ["ETA", "PAIRS", "N_GEN", "build_generators", "rotor_from_coefs",
+           "rotor_inverse", "eta_inner", "q_invariant"]

t3v3_wedge_memory.py

@@ -0,0 +1,173 @@
+"""Wedge associative memory — the geometric-algebra-native fix for cl33-opLM's
+key→value binding failure (MQAR recall ≈ 1/KV). See WEDGE_MEMORY_DESIGN.md.
+
+A rotor can only rotate the state; it cannot do an outer-product WRITE, which is how
+associative binding is stored (mLSTM: C += v kᵀ, read C q). The outer product in
+geometric algebra is the wedge, so we add a persistent leaky bivector memory:
+
+    M_t = γ_t · M_{t-1} + (K_t ∧ V_t)      (write; bivector, per block)
+    r_t = Q_t ⌋ M_{t-1}                     (read, before write — causal, no self-match)
+
+With the Cl(3,3) η-metric, the wedge/contraction reduce to a clean matrix form:
+    K∧V as matrix:  W = (V Kᵀ − K Vᵀ) · diag(η)      so that
+    Q ⌋ (K∧V) = W q = ⟨Q,K⟩_η V − ⟨Q,V⟩_η K
+i.e. when the query matches a stored key, the read returns that key's value.
+This is fast-weight / linear-attention associative memory, native to the algebra —
+the O(T) linear-recurrent sibling of the O(T²) operator-attention (same emitted Q/K).
+"""
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+
+
+class WedgeMemory(nn.Module):
+    """Per-block leaky bivector associative memory. Reuses the emitted query/key
+    rotors (via Rq, Rk applied to the state) — no new emission. Returns the read
+    r_t (grade-1, 6-d per block) to concatenate into the readout features.
+
+    Value = the transported state s_t (algebra-pure; keeps the operator-only
+    bottleneck / transparency — see design §5/§6 for the richer-V escalation)."""
+
+    def __init__(self, gate_bias_init: float = 3.0, transport: bool = True,
+                 key_source: str = "operator", n_gen: int = 15, induction: bool = True,
+                 d_model: int = None, nb: int = None, value_source: str = "same",
+                 delta_rule: bool = False):
+        super().__init__()
+        self.induction = induction   # key ← PREVIOUS token/operator (adjacency for MQAR)
+        self.key_source = key_source
+        # value_source: "same" = value from the same source as the key (default,
+        # legacy). "operator" = value is a learned 6-d projection of the emitted
+        # state-operator Bs (algebra-only, ablatable) — the G1 experiment: token KEY
+        # (clean context-free address) + ALGEBRA VALUE (LINEAR_TRANSPARENT_MEMORY.md).
+        self.value_source = value_source
+        self.delta_rule = delta_rule   # DeltaNet residual write: store v - M·k (reduces
+                                       # write interference by construction) — G/D3.
+        if value_source == "operator":
+            self.proj_v_op = nn.Linear(n_gen, 6)
+        self.nb = nb
+        # input-dependent forget gate γ_t = σ(w·x + b). bias_init 3.0 → γ ≈ 0.95.
+        # key_source options:
+        #   "operator": Q/K/V from emitted operators B_q/B_k/B_state (per-token but
+        #               CONTEXT-MIXED by the causal emitter → induction can't form a
+        #               clean key; falsified, recall ≈ 1/KV).
+        #   "state":    from the transported state (frame-entangled; + adjoint transport).
+        #   "token":    grade-1 Cl(3,3) projection of the RAW token embedding —
+        #               CONTEXT-FREE → clean induction key. Fully-transparent
+        #               (algebra-native η-wedge read) version of the token-copy win.
+        if key_source == "operator":
+            self.proj_q = nn.Linear(n_gen, 6); self.proj_k = nn.Linear(n_gen, 6)
+            self.proj_v = nn.Linear(n_gen, 6)
+            gate_in = n_gen
+        elif key_source == "token":
+            self.proj_q = nn.Linear(d_model, nb * 6); self.proj_k = nn.Linear(d_model, nb * 6)
+            self.proj_v = nn.Linear(d_model, nb * 6)
+            gate_in = d_model
+        else:
+            gate_in = 6
+        self.gate = nn.Linear(gate_in, 1)
+        nn.init.zeros_(self.gate.weight)
+        nn.init.constant_(self.gate.bias, gate_bias_init)
+        self.transport = transport and key_source == "state"
+
+    def forward(self, S, Rq, Rk, eta, R_state=None, Bs=None, Bq=None, Bk=None, tok_emb=None):
+        """Returns r (B,T,nb,6). fp32. operator: Bq/Bk/Bs (B,T,nb,15); state: S/Rq/Rk;
+        token: tok_emb (B,T,d_model) → grade-1 projections."""
+        eta = eta.float()
+        B, T, nb = S.shape[:3]
+        if self.key_source == "operator":
+            Q = self.proj_q(Bq.float()); K = self.proj_k(Bk.float())
+            V = self.proj_v(Bs.float())                  # per-token, context-mixed
+            gamma = torch.sigmoid(self.gate(Bs.float())).squeeze(-1)
+        elif self.key_source == "token":
+            te = tok_emb.float(); nb = self.nb
+            Q = self.proj_q(te).view(B, T, nb, 6)        # context-free grade-1 keys
+            K = self.proj_k(te).view(B, T, nb, 6)
+            if self.value_source == "operator":
+                # G1: algebra-only value — a learned projection of the emitted
+                # state-operator. Ablating operators (Bs=0) zeroes the value content
+                # → read dies → zero bypass (the property token-value lacked).
+                V = self.proj_v_op(Bs.float()).view(B, T, nb, 6)
+            else:
+                V = self.proj_v(te).view(B, T, nb, 6)    # token content (bypass)
+            gamma = torch.sigmoid(self.gate(te)).expand(B, T, nb)
+        else:
+            S = S.float(); Rq = Rq.float(); Rk = Rk.float()
+            Q = torch.einsum("btnij,btnj->btni", Rq, S)
+            K = torch.einsum("btnij,btnj->btni", Rk, S)
+            V = S
+            gamma = torch.sigmoid(self.gate(S)).squeeze(-1)
+        if self.induction:
+            # key at t ← the PREVIOUS token/operator → value_t stored under its
+            # predecessor, so a query retrieves what FOLLOWED it (MQAR adjacency).
+            K = torch.cat([torch.zeros_like(K[:, :1]), K[:, :-1]], dim=1)
+        etad = eta[None, None, :]                         # (1,1,6) for R⁻¹ = ηRᵀη
+        if self.transport and R_state is not None:
+            R_state = R_state.float()
+
+        M = torch.zeros(B, nb, 6, 6, device=S.device, dtype=torch.float32)
+        reads = []
+        for t in range(T):
+            if self.transport and R_state is not None:
+                Rt = R_state[:, t]
+                Rt_inv = etad[..., None] * Rt.transpose(-1, -2) * etad[..., None, :]
+                M = Rt @ M @ Rt_inv
+            q_t = Q[:, t]                                # (B,nb,6)
+            # READ before write (causal; avoids the query matching its own write)
+            reads.append(torch.einsum("bnij,bnj->bni", M, q_t))
+            # WRITE: W = (v_eff K_tᵀ − K_t v_effᵀ)·diag(η) = the bivector k∧v_eff as a matrix
+            v_t, k_t = V[:, t], K[:, t]
+            if self.delta_rule:
+                # DeltaNet residual: subtract what M already returns for THIS key, so the
+                # write corrects rather than clobbers (interference reduction by construction).
+                # STABILIZERS (2026-07-11): L2-normalize key (bounds ||M.k||) + beta write-gate
+                # in (0,1) => contraction guaranteed, fixes the M-blowup NaN.
+                k_t = k_t / (k_t.norm(dim=-1, keepdim=True) + 1e-6)
+                vpred = torch.einsum("bnij,bnj->bni", M, k_t)
+                v_eff = 0.5 * (v_t - vpred)
+            else:
+                v_eff = v_t
+            W = (torch.einsum("bni,bnj->bnij", v_eff, k_t)
+                 - torch.einsum("bni,bnj->bnij", k_t, v_eff)) * eta[None, None, None, :]
+            g = gamma[:, t][..., None, None]             # (B,nb,1,1)
+            M = g * M + W
+        return torch.stack(reads, dim=1)                 # (B,T,nb,6)
+
+
+class TokenCopyMemory(nn.Module):
+    """Minimal token-content value/copy channel — the measured transparency cost of
+    KV recall. A per-block fast-weight memory keyed on the RAW token embedding
+    (context-free AND position-free → the same token is the same key everywhere, so
+    it is matchable across positions and the value TOKEN can be copied). This
+    deliberately reads token content that bypasses the operator path; how much the
+    model leans on it (vs the operators) is the transparency price, measured directly.
+    Plain outer-product fast weights (token embeddings are not in the Cl(3,3) metric)."""
+
+    def __init__(self, d_model: int, nb: int, key_dim: int = 6, gate_bias_init: float = 3.0,
+                 induction: bool = True):
+        super().__init__()
+        self.nb, self.kd = nb, key_dim
+        self.induction = induction   # key ← PREVIOUS token (the shift that makes MQAR
+                                     # solvable: store value under the key that preceded it)
+        self.qkv = nn.Linear(d_model, nb * key_dim * 3)
+        self.gate = nn.Linear(d_model, nb)
+        nn.init.constant_(self.gate.bias, gate_bias_init)
+
+    def forward(self, tok_emb):
+        """tok_emb (B,T,d_model) — RAW token embedding (no pos). Returns r (B,T,nb,kd)."""
+        tok_emb = tok_emb.float()
+        B, T, _ = tok_emb.shape
+        qkv = self.qkv(tok_emb).view(B, T, self.nb, 3, self.kd)
+        q, k, v = qkv[..., 0, :], qkv[..., 1, :], qkv[..., 2, :]   # (B,T,nb,kd)
+        if self.induction:
+            # key at t ← the PREVIOUS token → value_t is stored under its predecessor,
+            # so querying with a token retrieves what FOLLOWED it (induction head).
+            k = torch.cat([torch.zeros_like(k[:, :1]), k[:, :-1]], dim=1)
+        gamma = torch.sigmoid(self.gate(tok_emb))                  # (B,T,nb)
+        M = torch.zeros(B, self.nb, self.kd, self.kd, device=tok_emb.device, dtype=torch.float32)
+        reads = []
+        for t in range(T):
+            reads.append(torch.einsum("bnij,bnj->bni", M, q[:, t]))  # read before write
+            outer = torch.einsum("bni,bnj->bnij", v[:, t], k[:, t])  # v ⊗ k
+            M = gamma[:, t][..., None, None] * M + outer
+        return torch.stack(reads, dim=1)                            # (B,T,nb,kd)

tape_memory.py

@@ -0,0 +1,180 @@
+"""Reversible tape memory — forward-readable exact memory native to the algebra.
+See REVERSIBLE_TAPE_DESIGN.md (2026-07-09).
+
+Design principle: TOKEN CONTENT MAY SELECT; ONLY ALGEBRA MAY FLOW.
+  - Addressing: context-free token keys (the wedge-validated mechanism) produce a
+    causal softmax α over past tape positions. Token content touches ONLY α.
+  - Read: r_t = Σ_i α_{t,i} · f(tape_i), where f yields pure algebra objects.
+
+The tape is the model's own reversible trace. The exact prefix products
+P_t = R_t···R_1 give the relative transport between any two positions:
+
+    R_{t←i} = P_t · P_i⁻¹,   with P_i⁻¹ = η P_iᵀ η   (exact, no replay)
+
+Degeneracy (design §3): transported absolute states are trivial (R_{t←i}s_i ∝ s_t),
+so the non-degenerate per-position content is exactly:
+  V1 "displacement": R_{t←i} applied to a fixed reference  → trajectory geometry only
+  V2 "state":        s_i raw (frame-mismatched on purpose)  → algebra-pure past state
+  V3 "increment":    B_i, the emitted bivector at i         → the local operator event
+                     (token-decodability ~82% per reversibility.py M2 → carries the
+                     content MQAR needs, without token embeddings in the value path)
+
+Op-ablation behavior (transparency mechanics): zeroed operators ⇒ R=I ⇒ P_t=I,
+displacement=const, increment=0, state=s0 — every value mode collapses while the
+token keys survive. The read CONTENT is operator-derived by construction.
+"""
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+VALUE_DIMS = {"displacement": 6, "state": 6, "increment": 15,
+              "multi": 21}  # displacement(6) + increment(15), one shared address
+
+
+class TapeMemory(nn.Module):
+    """Content-addressed read over the reversible tape. O(T²) score like the main
+    attention (fine at MQAR/32M scale; anchors+subsampling are the long-context
+    lever, not needed here)."""
+
+    def __init__(self, d_model: int, nb: int, value_mode: str = "increment",
+                 induction: bool = True, dual_address: bool = False, addr: str = "token"):
+        super().__init__()
+        assert value_mode in VALUE_DIMS, value_mode
+        assert addr in ("token", "operator", "target"), addr
+        self.nb, self.value_mode, self.induction, self.addr = nb, value_mode, induction, addr
+        # context-free token keys — addressing power proven by the wedge token mode
+        self.proj_q = nn.Linear(d_model, nb * 6)
+        self.proj_k = nn.Linear(d_model, nb * 6)
+        # v2.8 OPERATOR addressing (Garret): the address query is the LM's emitted OPERATOR, not the
+        # token. Keys = past operators. "what stored operator-combination fits what I'm computing now"
+        # instead of "what token matches" — the native relational address (op-dep memory), NOT the
+        # redundant token channel that CE routes around (v2.6 co-option). Ablating operators kills BOTH
+        # address and value → the whole tape becomes operator-derived (zero token bypass in addressing).
+        if addr == "operator":
+            self.proj_q_op = nn.Linear(nb * 15, nb * 6)
+            self.proj_k_op = nn.Linear(nb * 15, nb * 6)
+        # v2.8 TARGET-DRIVEN navigation (the diamond, facet 2): select by GOAL-vs-EFFECT fit, not
+        # key-match. target = learned "what I want" from the current operator; offer = each record's
+        # rotor ACTION on a learned reference (its composition effect). score = <target, offer>.
+        if addr == "target":
+            self.proj_target = nn.Linear(nb * 15, nb * 6)          # goal, from the current operator Bq
+            self.nav_ref = nn.Parameter(torch.randn(nb, 6) * 0.5)  # reference the record's rotor acts on
+        # v2.5 DUAL-ADDRESS (TAPE_ADDRESSING_V25.md): a token-FAITHFUL exact-match channel
+        # alongside the learned proj address. Probes showed the learned proj gets co-opted
+        # for LM context on natural text (attribution 0.98 raw-token → 0.006 learned); a
+        # dedicated raw-token-cosine address recovers ~0.98. exact_gate zero-init → a fresh
+        # v2.5 model is byte-identical to v2.4 at step 0, and v2.4 ckpts load unaffected.
+        self.dual_address = dual_address
+        if dual_address:
+            self.exact_tau = nn.Parameter(torch.tensor(8.0))     # sharp init (blockade-like)
+            self.exact_gate = nn.Parameter(torch.zeros(nb, 1))   # per-block, zero-init
+        # V1's fixed reference direction per block (constant → carries no content;
+        # the read then reflects trajectory displacement only). Only displacement-
+        # bearing modes use it — don't create an unused Parameter otherwise.
+        self.ref = (nn.Parameter(torch.randn(nb, 6) * 0.5)
+                    if value_mode in ("displacement", "multi") else None)
+
+    def out_dim(self) -> int:
+        return VALUE_DIMS[self.value_mode]
+
+    def forward(self, S, Bs, R_state, tok_emb, eta, Bq=None,
+                S_tape=None, Bs_tape=None, R_tape=None, tok_tape=None):
+        """S (B,T,nb,6) states; Bs (B,T,nb,15) emitted bivectors; R_state
+        (B,T,nb,6,6) per-step rotors; tok_emb (B,T,d_model) RAW token embedding
+        (addressing only). Returns r (B,T,nb,out_dim), fp32.
+
+        *_tape kwargs (SELF_STRUCTURE_CLAIMS.md, Claim-2 surgery): optional
+        COUNTERFACTUAL tape record — the read head sees these instead of the
+        factual history, while the live computation (scan/attention, and the
+        queries q) stays factual. Keys k and values are drawn from the tape
+        record (they are part of the record being counterfactually presented).
+        Absent kwargs = factual behavior, byte-identical to before."""
+        te = tok_emb.float()
+        B, T, nb = S.shape[:3]
+        S = S.float()
+        eta = eta.float()
+        # counterfactual record substitution (read-side only)
+        S_rec = S if S_tape is None else S_tape.float()
+        Bs_rec = Bs if Bs_tape is None else Bs_tape
+        R_rec = R_state if R_tape is None else R_tape
+        te_rec = te if tok_tape is None else tok_tape.float()
+
+        # --- addressing ---
+        # queries from the FACTUAL present; keys from the (possibly counterfactual) tape record.
+        # OPERATOR mode: address by the emitted operator (native relational key); token content
+        # never touches the address. TOKEN mode: the original context-free token key.
+        if self.addr == "target":
+            # goal-vs-effect fit (navigation, NOT key-match; no induction shift — this is not adjacency).
+            assert Bq is not None and R_rec is not None, "target addressing needs Bq and R_state (rotor of Bs)"
+            target = self.proj_target(Bq.float().reshape(B, T, nb * 15)).view(B, T, nb, 6)   # what I want
+            ref = self.nav_ref / self.nav_ref.norm(dim=-1, keepdim=True).clamp_min(1e-6)       # (nb,6)
+            offer = torch.einsum("btnij,nj->btni", R_rec.float(), ref)   # record EFFECT = rotor·ref (B,T,nb,6)
+            scores = torch.einsum("btni,bsni->bnts", target, offer) / (6 ** 0.5)   # <goal_t, effect_s>
+        else:
+            if self.addr == "operator":
+                assert Bq is not None, "operator addressing needs the query operator Bq"
+                q = self.proj_q_op(Bq.float().reshape(B, T, nb * 15)).view(B, T, nb, 6)
+                k = self.proj_k_op(Bs_rec.float().reshape(B, T, nb * 15)).view(B, T, nb, 6)
+            else:
+                q = self.proj_q(te).view(B, T, nb, 6)
+                k = self.proj_k(te_rec).view(B, T, nb, 6)
+            if self.induction:
+                # key at i ← the PREVIOUS token: querying with a key-token selects the position of what
+                # FOLLOWED it (MQAR adjacency, as in the wedge). Key-match only.
+                k = torch.cat([torch.zeros_like(k[:, :1]), k[:, :-1]], dim=1)
+            scores = torch.einsum("btni,bsni->bnts", q, k) / (6 ** 0.5)   # (B,nb,T,T)
+        causal = torch.triu(torch.ones(T, T, device=S.device, dtype=torch.bool), 0)
+        scores = scores.masked_fill(causal, float("-inf"))            # STRICT past (i<t)
+        A = F.softmax(scores, dim=-1)
+        A = torch.nan_to_num(A, nan=0.0)                              # t=0 has no past
+
+        # v2.5 DUAL-ADDRESS: token-FAITHFUL exact-match channel (raw token cosine, sharp).
+        # It only SELECTS (like A) — the value path is unchanged, so transparency holds.
+        A_exact = None
+        if self.dual_address:
+            tq = F.normalize(te, dim=-1)                              # (B,T,d) raw query token
+            tk = F.normalize(te_rec, dim=-1)                          # keys from the record
+            if self.induction:
+                tk = torch.cat([torch.zeros_like(tk[:, :1]), tk[:, :-1]], dim=1)
+            esc = torch.einsum("btd,bsd->bts", tq, tk) * self.exact_tau
+            esc = esc.masked_fill(causal[None], float("-inf"))
+            A_exact = torch.nan_to_num(F.softmax(esc, dim=-1), nan=0.0)[:, None]  # (B,1,T,T)
+
+        def _dual(r_ctx, value):
+            if A_exact is None:
+                return r_ctx
+            r_ex = torch.einsum("bnts,bsni->btni", A_exact.expand(B, nb, T, T), value)
+            return r_ctx + self.exact_gate * r_ex                     # gate zero-init → ≡ v2.4
+
+        # --- read (only algebra flows) ---
+        if self.value_mode == "state":
+            return _dual(torch.einsum("bnts,bsni->btni", A, S_rec), S_rec)
+        if self.value_mode == "increment":
+            return _dual(torch.einsum("bnts,bsni->btni", A, Bs_rec.float()), Bs_rec.float())
+        if self.value_mode == "multi":
+            # one α, two reads: increment (15) + displacement (6) — the local-order
+            # and global-path channels together (v2.4 design)
+            r_inc = _dual(torch.einsum("bnts,bsni->btni", A, Bs_rec.float()), Bs_rec.float())
+            r_disp = self._displacement_read(A, R_rec.float(), S.device, B, T, nb, eta)
+            return torch.cat([r_inc, r_disp], dim=-1)
+
+        # displacement: r_t = P_t · Σ_i α_{t,i} (P_i⁻¹ · ref̂)
+        return self._displacement_read(A, R_rec.float(), S.device, B, T, nb, eta)
+
+    def _displacement_read(self, A, R, device, B, T, nb, eta):
+        P = torch.empty(B, T, nb, 6, 6, device=device, dtype=torch.float32)
+        acc = torch.eye(6, device=device).expand(B, nb, 6, 6).contiguous()
+        for t in range(T):
+            acc = R[:, t] @ acc
+            P[:, t] = acc
+        ref = self.ref / self.ref.norm(dim=-1, keepdim=True).clamp_min(1e-6)  # (nb,6)
+        # P_i⁻¹ = η P_iᵀ η
+        Pinv = eta[None, None, None, :, None] * P.transpose(-1, -2) * eta[None, None, None, None, :]
+        u = torch.einsum("btnij,nj->btni", Pinv, ref)                 # (B,T,nb,6)
+        mix = torch.einsum("bnts,bsni->btni", A, u)
+        return torch.einsum("btnij,btnj->btni", P, mix)
+
+
+__all__ = ["TapeMemory", "VALUE_DIMS"]

wedge.py

@@ -0,0 +1,72 @@
+"""Exterior-product (wedge) tensors for the grade tower — metric-free.
+
+The lattice grade tower is g3=⟨B·v⟩₃, g4=⟨B·B_prev⟩₄, g5=⟨B·B_prev·v⟩₅,
+g6=⟨B·B_prev·B_prev2⟩₆. The top-grade part of a product of blades IS their
+wedge, and the wedge (exterior product) is METRIC-FREE — pure antisymmetrization.
+So the grade tower needs no Cayley/signature convention: fixed wedge tensors
+built from blade combinatorics, computed in PARALLEL per position (not in the
+sequential scan → no speed cost).
+
+Blade ordering: grade-k blade = sorted k-subset of {0..5} in `combinations`
+order. Grade-2 order matches so33.PAIRS (both are combinations(range(6),2)), so
+the model's emitted 15 bivector coefs align with these tensors directly.
+"""
+from __future__ import annotations
+
+from itertools import combinations
+
+import torch
+
+GRADE_DIM = {0: 1, 1: 6, 2: 15, 3: 20, 4: 15, 5: 6, 6: 1}
+
+
+def blades(grade):
+    return list(combinations(range(6), grade))
+
+
+def _shuffle_sign(A, B):
+    """Sign of wedge of sorted disjoint tuples A,B = (-1)^#{(a,b): a>b}."""
+    if set(A) & set(B):
+        return 0, None
+    inv = sum(1 for a in A for b in B if a > b)
+    merged = tuple(sorted(A + B))
+    return (-1) ** inv, merged
+
+
+def wedge_tensor(ga: int, gb: int) -> torch.Tensor:
+    """(dim_{ga+gb}, dim_ga, dim_gb): C[c] = Σ W[c,a,b]·A[a]·B[b] realizes A∧B."""
+    A, B, C = blades(ga), blades(gb), blades(ga + gb)
+    Ci = {b: i for i, b in enumerate(C)}
+    W = torch.zeros(len(C), len(A), len(B))
+    for ia, a in enumerate(A):
+        for ib, b in enumerate(B):
+            s, m = _shuffle_sign(a, b)
+            if s != 0:
+                W[Ci[m], ia, ib] = float(s)
+    return W
+
+
+class GradeTower(torch.nn.Module):
+    """Given per-position grade-1 state v and emitted grade-2 operators B (with
+    causal shifts B_prev, B_prev2), produce the grade tower features.
+
+    Inputs (all (..., dim)):  v (…,6), B (…,15)  [B_prev/B_prev2 are B shifted]
+    Output: dict of g2..g6 features. g1=v and g0 handled by caller.
+    """
+
+    def __init__(self):
+        super().__init__()
+        self.register_buffer("W_2_1", wedge_tensor(2, 1), persistent=False)  # B∧v -> g3
+        self.register_buffer("W_2_2", wedge_tensor(2, 2), persistent=False)  # B∧B -> g4
+        self.register_buffer("W_4_1", wedge_tensor(4, 1), persistent=False)  # g4∧v -> g5
+        self.register_buffer("W_4_2", wedge_tensor(4, 2), persistent=False)  # g4∧B2-> g6
+
+    def forward(self, v, B, B_prev, B_prev2):
+        g3 = torch.einsum("cab,...a,...b->...c", self.W_2_1, B, v)          # (...,20)
+        g4 = torch.einsum("cab,...a,...b->...c", self.W_2_2, B, B_prev)     # (...,15)
+        g5 = torch.einsum("cab,...a,...b->...c", self.W_4_1, g4, v)         # (...,6)
+        g6 = torch.einsum("cab,...a,...b->...c", self.W_4_2, g4, B_prev2)   # (...,1)
+        return g3, g4, g5, g6
+
+
+__all__ = ["GRADE_DIM", "wedge_tensor", "GradeTower"]