v0.1 paraphrase-invariance AR (model card + LoRA + linear_head + nla_meta)

README.md

@@ -0,0 +1,125 @@
+---
+license: cc-by-4.0
+language:
+  - en
+base_model: google/gemma-4-E2B
+tags:
+  - natural-language-autoencoder
+  - nla
+  - interpretability
+  - mechanistic-interpretability
+  - gemma
+  - consumer-gpu
+  - peft
+  - lora
+library_name: peft
+pipeline_tag: text-generation
+---
+
+# Gemma-4-E2B NLA AR (Critic) — v0.1 paraphrase-invariance retrain
+
+LoRA adapter for the AR (Critic) half of a Natural Language Autoencoder on Gemma-4-E2B, paired with `Solshine/gemma-4-e2b-nla-L23-av-v0_1_dd-step_250` for the v0.1 NLA pair release.
+
+This AR is a 50-step continuation from `Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1` trained with an auxiliary paraphrase-invariance loss:
+
+```
+L = MSE(AR(orig), gold) + lambda * MSE(AR(paraphrase), gold)
+```
+
+(where `lambda = 0.5` and `paraphrase` is a meaning-preserving rewrite of the explanation text). The original goal was to break the AR's content-blindness diagnosed in source-repo `FINDINGS.md §F72 Addendum 3`. The Phase A result was: the auxiliary loss **made the AR a better structural projector but not a content-reader**. See the "Findings" section below.
+
+## Training setup
+
+- Base: `google/gemma-4-E2B`
+- AR architecture: first K=18 layers of base + linear head (1536→1536, bias=True)
+- Quantization: NF4 4-bit base + bf16 LoRA
+- LoRA: r=64, alpha=128, target = language-model self-attn (q/k/v/o); inherited from v0.0.1
+- Loss head: linear_head.pt (saved alongside adapter)
+- Resume-from: v0.0.1 AR (`Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1`)
+- Optimizer: AdamW 8-bit, lr=5e-5 (lower than v0.0.1's 1e-4 — continuation training)
+- Batch: micro_batch=1, grad_accum=16 → effective batch 16
+- Steps: 50 SFT (~4.2h on the 4 GB GPU; ~5min/step due to doubled forward pass with paraphrase branch)
+- Max length: 512 tokens
+- Corpus: 696 rows of natural labeler-v1 / auditor-v2 paraphrase pairs from the persona+audit-haiku stage
+- Training script: `experiments/v8_nla_local/stage_ar_sft_v0_1.py` (source repo)
+
+## Findings (key)
+
+Three converging pieces of evidence for AR structural-projection on this hardware regime, documented in source-repo `FINDINGS.md §F72 Addendum 5`:
+
+1. **Loss-domain**: Phase A's `orig` and `para` losses tracked within 0.001-0.013 at *every* step. The v0.0.1 AR was already paraphrase-invariant in the loss domain before training began — the auxiliary `lambda·MSE(AR(paraphrase), gold)` term had nothing differential to attach to.
+
+2. **Direct delta_cos test (n=30)**:
+
+| Quantity | AR v0.0.1 | AR v0.1 |
+|---|---:|---:|
+| delta_cos = cos(AR(orig), gold) − cos(AR(para), gold) | +0.0008 | +0.0011 |
+| cos(AR(orig), gold) — absolute reconstruction | 0.4257 | 0.4966 (+17%) |
+| cos(AR(orig), AR(paraphrase)) | 0.9929 | 0.9948 |
+
+3. **H15 head-to-head (v0.1.dd step_250 AV, two ARs)**:
+
+| Input | AR v0.0.1 | AR v0.1 | absolute lift |
+|---|---:|---:|---:|
+| AV_OUT | 0.4221 | 0.4603 | +0.0382 |
+| EMPTY  | 0.4048 | 0.4531 | **+0.0483** |
+| RANDOM | 0.4045 | 0.4525 | **+0.0480** |
+| GIBBER | 0.4137 | 0.4562 | +0.0425 |
+| Δ(AV_OUT − EMPTY) | +0.0174 | +0.0072 | shrank |
+
+**Phase A's auxiliary loss uniformly lifted reconstruction quality across every input type** — including pure-noise EMPTY/RANDOM/GIBBER. AV_OUT got the *smallest* lift. The content-loaded delta shrank because the AR's "background structural projection" got more accurate faster than its content-loaded reading.
+
+The same pattern was re-confirmed on a second AV checkpoint (v0.1.dd step_50) — the structural-projection ceiling is upstream of AV training.
+
+## What this AR is good for
+
+- **Higher absolute round-trip cosine on the matched v0.1.dd AV**: 0.46 (vs 0.42 with v0.0.1 AR). Comparable to v0.0.1's published 0.438 ± 0.054.
+- **Cleaner structural-projection signature** for researchers studying the AR-side bottleneck. The fact that EMPTY/RANDOM/GIBBER lift MORE than AV_OUT in head-to-head is itself a publishable diagnostic.
+- **Methodology reproduction**: third-party researchers can re-run the Phase A pipeline on their own NLA training data.
+
+## What this AR does NOT do
+
+- Lift the H15 content-aware delta. The AR is still principally a structural projection at this scale.
+- Differentiate content-bearing AV outputs from gibberish input — the GIBBER vs AV_OUT gap is tiny (+0.004 on v0.1.dd step_250).
+
+## Loading
+
+```python
+import torch
+import torch.nn as nn
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+from huggingface_hub import snapshot_download
+
+bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16,
+                          bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True)
+base = AutoModelForCausalLM.from_pretrained("google/gemma-4-E2B", quantization_config=bnb,
+                                              device_map={"": torch.cuda.current_device()})
+ar_path = snapshot_download("Solshine/gemma-4-e2b-nla-L23-ar-v0_1-paraphrase-invariant")
+ar = PeftModel.from_pretrained(base, ar_path)
+head = nn.Linear(1536, 1536, bias=True).to(ar.device).to(torch.float32)
+head.load_state_dict(torch.load(f"{ar_path}/linear_head.pt", weights_only=True))
+```
+
+## See also
+
+- Matched AV (v0.1.dd): [`Solshine/gemma-4-e2b-nla-L23-av-v0_1_dd-step_250`](https://huggingface.co/Solshine/gemma-4-e2b-nla-L23-av-v0_1_dd-step_250)
+- v0.0.1 AR (baseline / resume-from): [`Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1`](https://huggingface.co/Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1)
+- v0.0.1 AV: [`Solshine/gemma-4-e2b-nla-L23-av-v0_0_1`](https://huggingface.co/Solshine/gemma-4-e2b-nla-L23-av-v0_0_1)
+- Public bundled repo: [`SolshineCode/nla-gemma-4-e2b`](https://github.com/SolshineCode/nla-gemma-4-e2b)
+
+## Citation
+
+```bibtex
+@misc{gemma4_e2b_nla_ar_v0_1,
+  title  = {Gemma-4-E2B NLA AR v0.1: paraphrase-invariance auxiliary loss + 3-piece-evidence for AR structural-projection at small-NLA scale},
+  author = {DeLeeuw, Caleb},
+  year   = {2026},
+  month  = {may},
+  url    = {https://huggingface.co/Solshine/gemma-4-e2b-nla-L23-ar-v0_1-paraphrase-invariant}
+}
+```
+
+Methodology citation:
+
+- Fraser-Taliente, K., et al. (2026). *Natural Language Autoencoders*. https://transformer-circuits.pub/2026/nla/

adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "google/gemma-4-E2B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "model\\.language_model\\.layers\\.\\d+\\.self_attn\\.(q_proj|k_proj|v_proj|o_proj)",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9dd6915b3fd7a798e230b1b530be979baaaf1ab93eab96906fb02851bbfc61e
+size 85750952

linear_head.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4470edf09dae0468699a5ac11ad60955497736be4aac7244110344afaaf6dbac
+size 9445317

nla_meta.yaml

@@ -0,0 +1,22 @@
+kind: nla_model
+role: critic
+ar_lora:
+  r: 64
+  alpha: 128
+  target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+critic:
+  num_hidden_layers: 18
+ar_head_dim_in: 1536
+ar_head_dim_out: 1536
+training:
+  version: v0.1_paraphrase_invariant
+  resume_from: C:\Users\caleb\AppData\Local\Temp\nla-gemma-4-e2b\weights\v0_0_1\ar
+  lr: 5.0e-05
+  max_steps: 50
+  micro_batch: 1
+  grad_accum: 16
+  para_lambda: 0.5