Publish trained weights, browser arrays and provenance manifest

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+web/bias.f32 filter=lfs diff=lfs merge=lfs -text
+web/gain.f32 filter=lfs diff=lfs merge=lfs -text
+web/head_a.f32 filter=lfs diff=lfs merge=lfs -text
+web/head_b.f32 filter=lfs diff=lfs merge=lfs -text
+web/in_proj.f32 filter=lfs diff=lfs merge=lfs -text
+web/ln_bias.f32 filter=lfs diff=lfs merge=lfs -text
+web/ln_weight.f32 filter=lfs diff=lfs merge=lfs -text
+web/rec_gain.f32 filter=lfs diff=lfs merge=lfs -text
+web/wte.f32 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,77 @@
+---
+license: cc-by-4.0
+language:
+  - en
+tags:
+  - connectomics
+  - drosophila
+  - connectome
+  - text-generation
+  - tiny-model
+datasets:
+  - fernandofernandes/fly-connectome-49k
+  - roneneldan/TinyStories
+pipeline_tag: text-generation
+---
+
+# Fly Wordbrain — rank 32
+
+**A 2,343,125-parameter language model on a frozen fruit-fly connectome — 22.5× smaller
+than the model it derives from, and still ahead of it on held-out text.**
+
+This is the smaller sibling of
+[fly-wordbrain-rank64](https://huggingface.co/fernandofernandes/fly-wordbrain-rank64),
+which is the recommended baseline. Everything about the architecture is identical except
+the readout rank: 49,393 → **32** → 1,024 instead of → 64 →.
+
+Research and receipts: [fly-wordbrain](https://github.com/fernando-neto-ai/fly-wordbrain)
+
+## What halving the readout again costs
+
+200 fresh TinyStories, 45,059 next-token targets, selecting no checkpoint.
+
+| Model | Parameters | Audit CE | Audit top-1 | ΔCE vs reference (95% CI) |
+|---|---:|---:|---:|---|
+| ngxson reference | 52,756,661 | 3.9882 | 31.38% | — |
+| rank 64 | 3,956,469 | 3.2802 | 33.37% | −0.708 [−0.738, −0.679] |
+| **rank 32 (this)** | **2,343,125** | **3.3091** | **32.58%** | **−0.679** [−0.711, −0.647] |
+
+Relative to rank 64: **1,613,344 fewer parameters** for **0.029 nats** and **0.79
+accuracy points**. On the selection-validation split that gap reads as 1.42 points, and
+its generated text has more visible failures on generic prompts. Take this one when size
+matters more than the last point of quality.
+
+| Component | Parameters |
+|---|---:|
+| Encoder (embedding + 8 input projections) | 482,816 |
+| Neuron gain, recurrent gain, bias | 148,179 |
+| Output LayerNorm | 98,786 |
+| Factorized readout (rank 32) | 1,613,344 |
+| **Total** | **2,343,125** |
+
+**No synaptic weight was modified.** The frozen-buffer digests in `manifest.json` are
+byte-identical to the reference model's and to rank 64's. Training changed the per-neuron
+dynamics and the two interfaces, nothing else.
+
+Both retained selectors land on update 16,600 here, so `min-ce.safetensors` and
+`max-accuracy.safetensors` hold the same weights under two different selection receipts.
+
+## Caveats
+
+The comparison against the released reference is **not** controlled — its training
+stories and trainer are unknown. The controlled finding is that constraining the readout
+rank acts as a regularizer against overfitting 1,000 short stories; it says nothing about
+whether the fly's wiring is a good prior for language. That would need randomized-graph
+and zero-edge controls, which have not been run.
+
+The connectome is not duplicated here; it lives in
+[fly-connectome-49k](https://huggingface.co/datasets/fernandofernandes/fly-connectome-49k).
+Loading instructions and the full step function are on the
+[rank-64 card](https://huggingface.co/fernandofernandes/fly-wordbrain-rank64).
+
+## Licence
+
+Weights **CC BY 4.0** — derived from MaleCNS v1.0 (FlyEM / HHMI Janelia, University of
+Cambridge, MRC LMB, Google Research) and the
+[`ngxson/fly-llm-hf`](https://huggingface.co/ngxson/fly-llm-hf) architecture. Code MIT.
+TinyStories is not redistributed.

experiment-config.json

@@ -0,0 +1,95 @@
+{
+  "schema_version": 1,
+  "configuration_role": "declarative experiment specification; CLI flags are authoritative",
+  "training_host": "macm3",
+  "device": "mps",
+  "reference_repository": "ngxson/fly-llm-hf",
+  "reference_revision": "65c677b3d566a2e9793d5f72999cdb441c6c0a9f",
+  "reference_weights_sha256": "355f06c44d14e38af50e9c801f51839c37a0a56ac4ca016da2be9b3ae6f215ad",
+  "dataset_path": "data/ngxson-tinystories-v1/dataset.json",
+  "dataset_sha256": "c1f553263545744786a8811ee6af7ace8c452820bbb99d15140b81625e40b3a8",
+  "split_stories": {
+    "train": 1000,
+    "validation": 100,
+    "test": 100
+  },
+  "initialization": "all learned values from scratch; retain released graph and interface",
+  "optimizer": {
+    "name": "AdamW",
+    "betas": [
+      0.9,
+      0.999
+    ],
+    "epsilon": 1e-08,
+    "weight_decay": 0.01,
+    "gradient_clip_norm": 1.0,
+    "phases": [
+      {
+        "epochs": 30,
+        "input_neuron_lr": 0.001,
+        "readout_lr": 0.0001
+      },
+      {
+        "epochs": 14,
+        "input_neuron_lr": 0.0003,
+        "readout_lr": 3e-05
+      }
+    ],
+    "schedule": "cosine to zero separately per phase; preserve Adam moments across phases"
+  },
+  "checkpoint_selection": {
+    "primary": "minimum token-weighted validation cross-entropy",
+    "secondary": "maximum validation top-1 accuracy; retain actual weights and selection receipt"
+  },
+  "claim_scope": "User-selected rank32 decoder comparison against preserved rank64; quality not yet measured",
+  "environment": {
+    "PYTORCH_ENABLE_MPS_FALLBACK": "0",
+    "OMP_NUM_THREADS": "4",
+    "VECLIB_MAXIMUM_THREADS": "4"
+  },
+  "experiment_id": "H32rank32fixed",
+  "branch": "exp/encoder32-readout32-fixed",
+  "trainer": "scripts/train_connectorch.py",
+  "connectorch_revision": "4bbfb645099aeb85bdbf850e1a87cc094769af87",
+  "groups_path": "data/connectorch-groups-v1/groups.npz",
+  "groups_sha256": "a07d311cd04126b5e54ea9cb3e8945470af9049bcb8499fb406e3c38d185dd7f",
+  "trainable_parameters": 2343125,
+  "training": {
+    "device": "mps",
+    "epochs": 30,
+    "second_epochs": 14,
+    "batch_size": 8,
+    "chunk_size": 32,
+    "seed": 42,
+    "eval_interval_updates": 100,
+    "d_embed": 32,
+    "plasticity": "fixed",
+    "readout_rank": 32,
+    "history_length": 8,
+    "skip_final_test": true
+  },
+  "edge_gain_parameters": 0,
+  "edge_multiplier_bounds": [
+    1.0,
+    1.0
+  ],
+  "neuron_gains": "original unconstrained gain and rec_gain",
+  "test_policy": "deferred until comparison and final protocol are fixed",
+  "output_relative": "results/connectorch-rank32-v1/arms/H32rank32fixed",
+  "readout_parameters": 1613344,
+  "encoder_parameters": 482816,
+  "comparison_baseline": "G32rank64fixed; compare common training target budgets and both independent validation selectors",
+  "readout_architecture": {
+    "input_neurons": 49393,
+    "rank": 32,
+    "vocabulary": 1024,
+    "bias": false,
+    "intermediate_activation": null
+  },
+  "initialization_note": "Same from-scratch trainer initialization and seed42; preserve encoder and neuron initialization. Rank32 changes readout shapes and initial logit variance relative to rank64; verify rank32-specific gradients and optimizer updates before training.",
+  "execution_status": "selected by user; separate rank32 native CPU/MPS preflight and optimizer smoke required",
+  "prerequisites": [
+    "Confirm idle macm3 and preserved G accepted stop with both winners/latest before H launch",
+    "Run separate rank32 CPU/MPS parity and eight-update optimizer smoke on macm3 before full H training"
+  ]
+}

manifest.json

@@ -0,0 +1,163 @@
+{
+  "schema_version": 1,
+  "created_at_utc": "2026-09-16T21:54:24.705349+00:00",
+  "experiment": "H32rank32fixed",
+  "architecture": {
+    "neurons": 49393,
+    "edges": 9050172,
+    "delay_slots": 8,
+    "encoder_width": 32,
+    "readout_rank": 32,
+    "vocabulary": 1024,
+    "edge_adaptation": "fixed",
+    "recurrence": "x = 0.1*x + 0.9*tanh(gain*(rec_gain*(W@x) + drive) + bias)"
+  },
+  "trainable_parameters": 2343125,
+  "encoder_parameters": 482816,
+  "readout_parameters": 1613344,
+  "graph": {
+    "dataset": "fernandofernandes/fly-connectome-49k",
+    "note": "The connectome is not duplicated here. The frozen-buffer digests below identify the exact graph these weights were trained on; the dataset manifest carries the same digests."
+  },
+  "selectors": {
+    "minimum_validation_ce": {
+      "file": "min-ce.safetensors",
+      "selector": "minimum_validation_ce",
+      "updates": 16600,
+      "source_checkpoint_sha256": "6b9badc7fb43d883b50ff3538601b333690317cab572a7c90619c83633cf0515",
+      "safetensors_sha256": "bdbde3e3018f9ef7549dc604d5d89dfe7c7945fa71a44605f7a4c9a21753ff2a",
+      "validation": {
+        "cross_entropy": 3.1376539814501627,
+        "top1_accuracy": 0.354713358324952,
+        "correct": 7759,
+        "tokens": 21874,
+        "stories": 100
+      },
+      "frozen_buffers_sha256": {
+        "brain.w_offsets": "53b216d7bc2503c539c2d2b03ce56f66eb8a2e9b902f0a20deda0ec4be37ade0",
+        "brain.w_indices": "0f17b9563457fbf84404c54227db86b5adcf04a689cdff5e33d32ec5d5e0b771",
+        "brain.w_values": "e6408887d3312783c65d3f10bb69b28fe50bb06699f0a8a9341d444e6eb0549a",
+        "brain.in_index": "ee440072f6c94b59c552cb09812af5894066ae28f686cda2cd2e35a3626beda6",
+        "brain.out_index": "5ac16e2c38cf0a1b4b4aea74eadeb2d487e153113d046895a813e09d010f0a18",
+        "brain.edge_group_index": "4b9af4b1e2313aba5a9a139ae3ecfc8fcae09a440c65802207a26775700f6eca",
+        "brain.node_type_index": "9ea440846c4bfde95c8c007320ad3bc3b79cfb3b69c2ff6a136c39fd41f658be"
+      },
+      "parameters": 2343125
+    },
+    "maximum_validation_accuracy": {
+      "file": "max-accuracy.safetensors",
+      "selector": "maximum_validation_accuracy",
+      "updates": 16600,
+      "source_checkpoint_sha256": "6ed28abd44ec3d7049573b6ebdc3b81723991da20def653b23b3e15af001b2c8",
+      "safetensors_sha256": "8b50b6da2b48de90bf21508dcd4cad53e9e1bbbb2373feb97571036fb97492f5",
+      "validation": {
+        "cross_entropy": 3.1376539814501627,
+        "top1_accuracy": 0.354713358324952,
+        "correct": 7759,
+        "tokens": 21874,
+        "stories": 100
+      },
+      "frozen_buffers_sha256": {
+        "brain.w_offsets": "53b216d7bc2503c539c2d2b03ce56f66eb8a2e9b902f0a20deda0ec4be37ade0",
+        "brain.w_indices": "0f17b9563457fbf84404c54227db86b5adcf04a689cdff5e33d32ec5d5e0b771",
+        "brain.w_values": "e6408887d3312783c65d3f10bb69b28fe50bb06699f0a8a9341d444e6eb0549a",
+        "brain.in_index": "ee440072f6c94b59c552cb09812af5894066ae28f686cda2cd2e35a3626beda6",
+        "brain.out_index": "5ac16e2c38cf0a1b4b4aea74eadeb2d487e153113d046895a813e09d010f0a18",
+        "brain.edge_group_index": "4b9af4b1e2313aba5a9a139ae3ecfc8fcae09a440c65802207a26775700f6eca",
+        "brain.node_type_index": "9ea440846c4bfde95c8c007320ad3bc3b79cfb3b69c2ff6a136c39fd41f658be"
+      },
+      "parameters": 2343125
+    }
+  },
+  "held_out_scores": {
+    "released-reference": {
+      "parameters": 52756661,
+      "audit": {
+        "stories": 200,
+        "tokens": 45059,
+        "nll_sum": 179705.70758149194,
+        "correct": 14141,
+        "cross_entropy": 3.9882311542975195,
+        "perplexity": 53.959359118617655,
+        "top1_accuracy": 0.3138329745444861
+      },
+      "validation": {
+        "stories": 100,
+        "tokens": 21874,
+        "nll_sum": 80516.69813330428,
+        "correct": 7709,
+        "cross_entropy": 3.6809316143962825,
+        "perplexity": 39.68334643405466,
+        "top1_accuracy": 0.3524275395446649
+      }
+    },
+    "H32rank32fixed/min-ce": {
+      "parameters": 2343125,
+      "audit": {
+        "stories": 200,
+        "tokens": 45059,
+        "nll_sum": 149104.48977598827,
+        "correct": 14680,
+        "cross_entropy": 3.3090945155460236,
+        "perplexity": 27.36033988969138,
+        "top1_accuracy": 0.32579506868772057
+      },
+      "validation": {
+        "stories": 100,
+        "tokens": 21874,
+        "nll_sum": 68633.04354491424,
+        "correct": 7759,
+        "cross_entropy": 3.137653997664544,
+        "perplexity": 23.049728662111555,
+        "top1_accuracy": 0.354713358324952
+      }
+    }
+  },
+  "training": {
+    "reference": "ngxson/fly-llm-hf",
+    "reference_revision": "65c677b3d566a2e9793d5f72999cdb441c6c0a9f",
+    "dataset": "TinyStories, 1,000 training stories (not redistributed)",
+    "dataset_sha256": "c1f553263545744786a8811ee6af7ace8c452820bbb99d15140b81625e40b3a8",
+    "seed": 42,
+    "host": "Apple M3 Max, PyTorch MPS",
+    "stop": "accepted early stop under a plateau rule; the planned 44-epoch schedule was not completed"
+  },
+  "web_export": {
+    "files": [
+      "wte",
+      "in_proj",
+      "gain",
+      "rec_gain",
+      "bias",
+      "ln_weight",
+      "ln_bias",
+      "head_a",
+      "head_b"
+    ],
+    "config": {
+      "neurons": 49393,
+      "delay_k": 8,
+      "vocabulary": 1024,
+      "d_embed": 32,
+      "readout_rank": 32,
+      "neurons_per_slot": 1758,
+      "leak": 0.9,
+      "pad_token_id": 0,
+      "bos_token_id": 1,
+      "eos_token_id": 2,
+      "layer_norm_eps": 1e-05
+    },
+    "golden_prompts": [
+      "Once upon a time, there was a",
+      "One day, a little girl named Lily found a needle",
+      "Tom and his dog",
+      "The little bird was afraid of the rain. Its mother"
+    ],
+    "note": "Raw little-endian float32 arrays streamed by the browser demo. The manifest's golden trace was verified against PyTorch token-for-token."
+  },
+  "license": {
+    "code": "MIT",
+    "weights": "CC-BY-4.0",
+    "attribution": "Derived from the MaleCNS v1.0 connectome (FlyEM / HHMI Janelia, University of Cambridge, MRC LMB, Google Research) and the ngxson/fly-llm-hf architecture."
+  }
+}

max-accuracy.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b50b6da2b48de90bf21508dcd4cad53e9e1bbbb2373feb97571036fb97492f5
+size 9373436

min-ce.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bdbde3e3018f9ef7549dc604d5d89dfe7c7945fa71a44605f7a4c9a21753ff2a
+size 9373428

web/bias.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1eb8cbf51cf6072eed5fcc15a649c55d3fe57590e9fc21f8b8f386b4b9dcb63a
+size 197572

web/gain.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc672b1ab45bf4501fb0b5a56c59360de459a6d4d1db7f46d3efe5d1c241a319
+size 197572

web/head_a.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:166a967c635c3390807ca6426f7976e82c279eb39924f08fa7e4074e93c41e90
+size 6322304

web/head_b.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb343c955dffdea7539e1312b456c78516eaecfb9e7c27a2d513baf4b9644570
+size 131072

web/in_proj.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5dc2dbb50b7ac72a216ecce115751c6e4602f345904c495efa63f7afad884c9
+size 1800192

web/ln_bias.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67f612750c4d487236091d8538876c54c73bae9462ac0bc05293367f8cec0098
+size 197572

web/ln_weight.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f81f287b5abb97047bc7182d6e4fa5d070f80a97323c274dd499378dc8d11f2c
+size 197572

web/manifest.json

@@ -0,0 +1,1533 @@
+{
+  "schema_version": 1,
+  "created_at_utc": "2026-09-16T21:39:08.574521+00:00",
+  "experiment": "H32rank32fixed",
+  "selector": "H32rank32fixed-best",
+  "checkpoint_sha256": "6b9badc7fb43d883b50ff3538601b333690317cab572a7c90619c83633cf0515",
+  "updates": 16600,
+  "trainable_parameters": 2343125,
+  "config": {
+    "neurons": 49393,
+    "delay_k": 8,
+    "vocabulary": 1024,
+    "d_embed": 32,
+    "readout_rank": 32,
+    "neurons_per_slot": 1758,
+    "leak": 0.9,
+    "pad_token_id": 0,
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "layer_norm_eps": 1e-05
+  },
+  "index_checks": {
+    "in_index_unique": true,
+    "out_index_is_identity": true
+  },
+  "graph": {
+    "dataset": "fernandofernandes/fly-connectome-49k",
+    "edges": 9050172,
+    "encoding": "graph/edges_weight.f32 (exact) with graph/edges_source.u16 and graph/edges_offsets.i32",
+    "quantised_alternative": {
+      "file": "graph/edges_weight.i16",
+      "scale": 1.1509252714347508e-05,
+      "halves_the_edge_payload": true
+    },
+    "frozen_buffers_sha256": {
+      "brain.w_offsets": "53b216d7bc2503c539c2d2b03ce56f66eb8a2e9b902f0a20deda0ec4be37ade0",
+      "brain.w_indices": "0f17b9563457fbf84404c54227db86b5adcf04a689cdff5e33d32ec5d5e0b771",
+      "brain.w_values": "e6408887d3312783c65d3f10bb69b28fe50bb06699f0a8a9341d444e6eb0549a",
+      "brain.in_index": "ee440072f6c94b59c552cb09812af5894066ae28f686cda2cd2e35a3626beda6",
+      "brain.out_index": "5ac16e2c38cf0a1b4b4aea74eadeb2d487e153113d046895a813e09d010f0a18",
+      "brain.edge_group_index": "4b9af4b1e2313aba5a9a139ae3ecfc8fcae09a440c65802207a26775700f6eca",
+      "brain.node_type_index": "9ea440846c4bfde95c8c007320ad3bc3b79cfb3b69c2ff6a136c39fd41f658be"
+    }
+  },
+  "layout": {
+    "in_index": "interface/in_index.i32 from the connectome dataset",
+    "out_index": "identity",
+    "drive": "concatenate wte[ext[k-j]] @ in_proj[j] for j in 0..k-1, then index_add at in_index",
+    "recurrence": "x = (1-leak)*x + leak*tanh(gain*(rec_gain*(W@x) + drive) + bias)",
+    "readout": "logits = layer_norm(x) @ head_a.T @ head_b.T"
+  },
+  "files": {
+    "wte": {
+      "file": "wte.f32",
+      "shape": [
+        1024,
+        32
+      ],
+      "dtype": "float32",
+      "bytes": 131072,
+      "sha256": "d2ce3cc6a0d92633b2c0de0e22d8fc2545369c0f00bab28b6366f698d2458ba8"
+    },
+    "in_proj": {
+      "file": "in_proj.f32",
+      "shape": [
+        8,
+        32,
+        1758
+      ],
+      "dtype": "float32",
+      "bytes": 1800192,
+      "sha256": "d5dc2dbb50b7ac72a216ecce115751c6e4602f345904c495efa63f7afad884c9"
+    },
+    "gain": {
+      "file": "gain.f32",
+      "shape": [
+        49393
+      ],
+      "dtype": "float32",
+      "bytes": 197572,
+      "sha256": "cc672b1ab45bf4501fb0b5a56c59360de459a6d4d1db7f46d3efe5d1c241a319"
+    },
+    "rec_gain": {
+      "file": "rec_gain.f32",
+      "shape": [
+        49393
+      ],
+      "dtype": "float32",
+      "bytes": 197572,
+      "sha256": "29250b4f7d8f2d036ec7e95207dcc67384868886cafc62a9d9f5c116e1bfa594"
+    },
+    "bias": {
+      "file": "bias.f32",
+      "shape": [
+        49393
+      ],
+      "dtype": "float32",
+      "bytes": 197572,
+      "sha256": "1eb8cbf51cf6072eed5fcc15a649c55d3fe57590e9fc21f8b8f386b4b9dcb63a"
+    },
+    "ln_weight": {
+      "file": "ln_weight.f32",
+      "shape": [
+        49393
+      ],
+      "dtype": "float32",
+      "bytes": 197572,
+      "sha256": "f81f287b5abb97047bc7182d6e4fa5d070f80a97323c274dd499378dc8d11f2c"
+    },
+    "ln_bias": {
+      "file": "ln_bias.f32",
+      "shape": [
+        49393
+      ],
+      "dtype": "float32",
+      "bytes": 197572,
+      "sha256": "67f612750c4d487236091d8538876c54c73bae9462ac0bc05293367f8cec0098"
+    },
+    "head_a": {
+      "file": "head_a.f32",
+      "shape": [
+        32,
+        49393
+      ],
+      "dtype": "float32",
+      "bytes": 6322304,
+      "sha256": "166a967c635c3390807ca6426f7976e82c279eb39924f08fa7e4074e93c41e90"
+    },
+    "head_b": {
+      "file": "head_b.f32",
+      "shape": [
+        1024,
+        32
+      ],
+      "dtype": "float32",
+      "bytes": 131072,
+      "sha256": "bb343c955dffdea7539e1312b456c78516eaecfb9e7c27a2d513baf4b9644570"
+    }
+  },
+  "verification": {
+    "numpy_matches_torch": [
+      {
+        "prompt": "Once upon a time, there was a",
+        "numpy_tokens": [
+          364,
+          413,
+          485,
+          448,
+          16,
+          329,
+          490,
+          269,
+          372,
+          346,
+          310,
+          842
+        ],
+        "torch_tokens": [
+          364,
+          413,
+          485,
+          448,
+          16,
+          329,
+          490,
+          269,
+          372,
+          346,
+          310,
+          842
+        ],
+        "identical": true
+      },
+      {
+        "prompt": "One day, a little girl named Lily found a needle",
+        "numpy_tokens": [
+          16,
+          329,
+          277,
+          935,
+          880,
+          85,
+          597,
+          271,
+          321,
+          490,
+          269,
+          372
+        ],
+        "torch_tokens": [
+          16,
+          329,
+          277,
+          935,
+          880,
+          85,
+          597,
+          271,
+          321,
+          490,
+          269,
+          372
+        ],
+        "identical": true
+      }
+    ],
+    "quantisation_drift": [
+      {
+        "prompt": "Once upon a time, there was a",
+        "exact_tokens": 60,
+        "quantised_tokens": 60,
+        "identical": true,
+        "first_divergent_step": null
+      },
+      {
+        "prompt": "One day, a little girl named Lily found a needle",
+        "exact_tokens": 60,
+        "quantised_tokens": 60,
+        "identical": false,
+        "first_divergent_step": 32
+      },
+      {
+        "prompt": "Tom and his dog",
+        "exact_tokens": 19,
+        "quantised_tokens": 19,
+        "identical": true,
+        "first_divergent_step": null
+      },
+      {
+        "prompt": "The little bird was afraid of the rain. Its mother",
+        "exact_tokens": 60,
+        "quantised_tokens": 60,
+        "identical": true,
+        "first_divergent_step": null
+      }
+    ],
+    "note": "Golden traces use the exact float32 graph the browser downloads, so a JavaScript mismatch indicates an implementation defect. The drift rows record what the smaller int16 edge file would change instead."
+  },
+  "golden": [
+    {
+      "prompt": "Once upon a time, there was a",
+      "prompt_ids": [
+        1,
+        393,
+        417,
+        261,
+        368,
+        14,
+        370,
+        277,
+        261
+      ],
+      "tokens": [
+        364,
+        413,
+        485,
+        448,
+        16,
+        329,
+        490,
+        269,
+        372,
+        346,
+        310,
+        842,
+        271,
+        372,
+        346,
+        310,
+        842,
+        16,
+        493,
+        339,
+        14,
+        448,
+        385,
+        380,
+        495,
+        310,
+        269,
+        310,
+        676,
+        16,
+        329,
+        277,
+        381,
+        398,
+        269,
+        545,
+        496,
+        16,
+        201,
+        201,
+        786,
+        277,
+        381,
+        398,
+        269,
+        545,
+        269,
+        264,
+        574,
+        346,
+        310,
+        676,
+        16,
+        329,
+        604,
+        310,
+        380,
+        14,
+        348,
+        47
+      ],
+      "text": "Once upon a time, there was a little girl named Lily. She loved to play with her toys and play with her toys. One day, Lily's mom asked her to her mommy. She was very happy to have fun.\n\nLily was very happy to have to the park with her mommy. She put her mom, \"M",
+      "continuation": " little girl named Lily. She loved to play with her toys and play with her toys. One day, Lily's mom asked her to her mommy. She was very happy to have fun.\n\nLily was very happy to have to the park with her mommy. She put her mom, \"M",
+      "steps": [
+        {
+          "token": 364,
+          "top1_probability": 0.5586004257202148,
+          "mean_abs_state": 0.24818630516529083
+        },
+        {
+          "token": 413,
+          "top1_probability": 0.6323819756507874,
+          "mean_abs_state": 0.24588727951049805
+        },
+        {
+          "token": 485,
+          "top1_probability": 0.8617414236068726,
+          "mean_abs_state": 0.24254217743873596
+        },
+        {
+          "token": 448,
+          "top1_probability": 0.877619743347168,
+          "mean_abs_state": 0.24094882607460022
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.9414868354797363,
+          "mean_abs_state": 0.24149666726589203
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.7390103936195374,
+          "mean_abs_state": 0.24070635437965393
+        },
+        {
+          "token": 490,
+          "top1_probability": 0.7238146066665649,
+          "mean_abs_state": 0.24336165189743042
+        },
+        {
+          "token": 269,
+          "top1_probability": 0.826725959777832,
+          "mean_abs_state": 0.24224962294101715
+        },
+        {
+          "token": 372,
+          "top1_probability": 0.699946403503418,
+          "mean_abs_state": 0.24070605635643005
+        },
+        {
+          "token": 346,
+          "top1_probability": 0.507792055606842,
+          "mean_abs_state": 0.23992347717285156
+        },
+        {
+          "token": 310,
+          "top1_probability": 0.8137639760971069,
+          "mean_abs_state": 0.24025946855545044
+        },
+        {
+          "token": 842,
+          "top1_probability": 0.8370879888534546,
+          "mean_abs_state": 0.24348165094852448
+        },
+        {
+          "token": 271,
+          "top1_probability": 0.3544219434261322,
+          "mean_abs_state": 0.24309156835079193
+        },
+        {
+          "token": 372,
+          "top1_probability": 0.146468386054039,
+          "mean_abs_state": 0.2437543272972107
+        },
+        {
+          "token": 346,
+          "top1_probability": 0.3894433379173279,
+          "mean_abs_state": 0.2426215261220932
+        },
+        {
+          "token": 310,
+          "top1_probability": 0.6416600942611694,
+          "mean_abs_state": 0.2410142570734024
+        },
+        {
+          "token": 842,
+          "top1_probability": 0.4848432242870331,
+          "mean_abs_state": 0.24306263029575348
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.7696483135223389,
+          "mean_abs_state": 0.24520659446716309
+        },
+        {
+          "token": 493,
+          "top1_probability": 0.9733051061630249,
+          "mean_abs_state": 0.24622918665409088
+        },
+        {
+          "token": 339,
+          "top1_probability": 0.9913354516029358,
+          "mean_abs_state": 0.24643437564373016
+        },
+        {
+          "token": 14,
+          "top1_probability": 0.9551123976707458,
+          "mean_abs_state": 0.24343281984329224
+        },
+        {
+          "token": 448,
+          "top1_probability": 0.47521740198135376,
+          "mean_abs_state": 0.2403506487607956
+        },
+        {
+          "token": 385,
+          "top1_probability": 0.5366150736808777,
+          "mean_abs_state": 0.24467521905899048
+        },
+        {
+          "token": 380,
+          "top1_probability": 0.44833260774612427,
+          "mean_abs_state": 0.23982226848602295
+        },
+        {
+          "token": 495,
+          "top1_probability": 0.17207364737987518,
+          "mean_abs_state": 0.2383100837469101
+        },
+        {
+          "token": 310,
+          "top1_probability": 0.8984615206718445,
+          "mean_abs_state": 0.23752246797084808
+        },
+        {
+          "token": 269,
+          "top1_probability": 0.5566059350967407,
+          "mean_abs_state": 0.23732325434684753
+        },
+        {
+          "token": 310,
+          "top1_probability": 0.2827357053756714,
+          "mean_abs_state": 0.23743946850299835
+        },
+        {
+          "token": 676,
+          "top1_probability": 0.25556260347366333,
+          "mean_abs_state": 0.23878537118434906
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.15785269439220428,
+          "mean_abs_state": 0.23971794545650482
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.4161810576915741,
+          "mean_abs_state": 0.24508832395076752
+        },
+        {
+          "token": 277,
+          "top1_probability": 0.07454168051481247,
+          "mean_abs_state": 0.2462906688451767
+        },
+        {
+          "token": 381,
+          "top1_probability": 0.1884118765592575,
+          "mean_abs_state": 0.24663342535495758
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.28206750750541687,
+          "mean_abs_state": 0.24427202343940735
+        },
+        {
+          "token": 269,
+          "top1_probability": 0.4160069525241852,
+          "mean_abs_state": 0.24202421307563782
+        },
+        {
+          "token": 545,
+          "top1_probability": 0.30667850375175476,
+          "mean_abs_state": 0.2402380108833313
+        },
+        {
+          "token": 496,
+          "top1_probability": 0.1465916931629181,
+          "mean_abs_state": 0.24212458729743958
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.36332035064697266,
+          "mean_abs_state": 0.24332304298877716
+        },
+        {
+          "token": 201,
+          "top1_probability": 0.17865613102912903,
+          "mean_abs_state": 0.2410154789686203
+        },
+        {
+          "token": 201,
+          "top1_probability": 0.9879697561264038,
+          "mean_abs_state": 0.24228864908218384
+        },
+        {
+          "token": 786,
+          "top1_probability": 0.15415850281715393,
+          "mean_abs_state": 0.24302853643894196
+        },
+        {
+          "token": 277,
+          "top1_probability": 0.15501132607460022,
+          "mean_abs_state": 0.2390313297510147
+        },
+        {
+          "token": 381,
+          "top1_probability": 0.19568607211112976,
+          "mean_abs_state": 0.23789797723293304
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.3770677447319031,
+          "mean_abs_state": 0.23960325121879578
+        },
+        {
+          "token": 269,
+          "top1_probability": 0.3582480847835541,
+          "mean_abs_state": 0.24009329080581665
+        },
+        {
+          "token": 545,
+          "top1_probability": 0.1858958899974823,
+          "mean_abs_state": 0.23879685997962952
+        },
+        {
+          "token": 269,
+          "top1_probability": 0.17902468144893646,
+          "mean_abs_state": 0.24035723507404327
+        },
+        {
+          "token": 264,
+          "top1_probability": 0.41966506838798523,
+          "mean_abs_state": 0.24132762849330902
+        },
+        {
+          "token": 574,
+          "top1_probability": 0.09898614138364792,
+          "mean_abs_state": 0.24146541953086853
+        },
+        {
+          "token": 346,
+          "top1_probability": 0.24348269402980804,
+          "mean_abs_state": 0.24259431660175323
+        },
+        {
+          "token": 310,
+          "top1_probability": 0.4910120964050293,
+          "mean_abs_state": 0.24335317313671112
+        },
+        {
+          "token": 676,
+          "top1_probability": 0.13613320887088776,
+          "mean_abs_state": 0.24214254319667816
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.7580757141113281,
+          "mean_abs_state": 0.24119961261749268
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.45265501737594604,
+          "mean_abs_state": 0.24390295147895813
+        },
+        {
+          "token": 604,
+          "top1_probability": 0.05739033222198486,
+          "mean_abs_state": 0.24665330350399017
+        },
+        {
+          "token": 310,
+          "top1_probability": 0.2039330005645752,
+          "mean_abs_state": 0.24261923134326935
+        },
+        {
+          "token": 380,
+          "top1_probability": 0.13538669049739838,
+          "mean_abs_state": 0.24215972423553467
+        },
+        {
+          "token": 14,
+          "top1_probability": 0.16917967796325684,
+          "mean_abs_state": 0.2429213523864746
+        },
+        {
+          "token": 348,
+          "top1_probability": 0.2981824278831482,
+          "mean_abs_state": 0.2438736855983734
+        },
+        {
+          "token": 47,
+          "top1_probability": 0.11140884459018707,
+          "mean_abs_state": 0.24439094960689545
+        }
+      ],
+      "final_state_sha256": "216fa0ed779a457850ae5ee574c5630617b732bdeedcdbd0aad41c37f0a58009",
+      "final_state_mean_abs": 0.24418696761131287
+    },
+    {
+      "prompt": "One day, a little girl named Lily found a needle",
+      "prompt_ids": [
+        1,
+        404,
+        339,
+        14,
+        261,
+        364,
+        413,
+        485,
+        448,
+        557,
+        261,
+        835,
+        299
+      ],
+      "tokens": [
+        16,
+        329,
+        277,
+        935,
+        880,
+        85,
+        597,
+        271,
+        321,
+        490,
+        269,
+        372,
+        16,
+        493,
+        339,
+        14,
+        321,
+        418,
+        261,
+        406,
+        14,
+        321,
+        418,
+        261,
+        406,
+        638,
+        16,
+        329,
+        277,
+        381,
+        398,
+        271,
+        335,
+        14,
+        348,
+        43,
+        305,
+        887,
+        383,
+        536,
+        372,
+        346,
+        771,
+        474,
+        16,
+        567,
+        71,
+        536,
+        420,
+        322,
+        969,
+        14,
+        411,
+        304,
+        385,
+        483,
+        322,
+        969,
+        470,
+        315
+      ],
+      "text": "One day, a little girl named Lily found a needle. She was three years old and she loved to play. One day, she saw a big, she saw a big tree. She was very happy and said, \"I think you can play with my friends. We can't be careful, but it's not be careful.\" The",
+      "continuation": ". She was three years old and she loved to play. One day, she saw a big, she saw a big tree. She was very happy and said, \"I think you can play with my friends. We can't be careful, but it's not be careful.\" The",
+      "steps": [
+        {
+          "token": 16,
+          "top1_probability": 0.21929176151752472,
+          "mean_abs_state": 0.24182972311973572
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.6597888469696045,
+          "mean_abs_state": 0.242539644241333
+        },
+        {
+          "token": 277,
+          "top1_probability": 0.334426611661911,
+          "mean_abs_state": 0.24500830471515656
+        },
+        {
+          "token": 935,
+          "top1_probability": 0.1496257185935974,
+          "mean_abs_state": 0.24282312393188477
+        },
+        {
+          "token": 880,
+          "top1_probability": 0.7430214881896973,
+          "mean_abs_state": 0.24047231674194336
+        },
+        {
+          "token": 85,
+          "top1_probability": 0.8860323429107666,
+          "mean_abs_state": 0.23745857179164886
+        },
+        {
+          "token": 597,
+          "top1_probability": 0.8477005958557129,
+          "mean_abs_state": 0.23506495356559753
+        },
+        {
+          "token": 271,
+          "top1_probability": 0.764802873134613,
+          "mean_abs_state": 0.2364216297864914
+        },
+        {
+          "token": 321,
+          "top1_probability": 0.15685617923736572,
+          "mean_abs_state": 0.23958107829093933
+        },
+        {
+          "token": 490,
+          "top1_probability": 0.22659173607826233,
+          "mean_abs_state": 0.2377837896347046
+        },
+        {
+          "token": 269,
+          "top1_probability": 0.514065146446228,
+          "mean_abs_state": 0.2385009527206421
+        },
+        {
+          "token": 372,
+          "top1_probability": 0.22299788892269135,
+          "mean_abs_state": 0.23995035886764526
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.4297165274620056,
+          "mean_abs_state": 0.24061356484889984
+        },
+        {
+          "token": 493,
+          "top1_probability": 0.30934134125709534,
+          "mean_abs_state": 0.24385303258895874
+        },
+        {
+          "token": 339,
+          "top1_probability": 0.927859902381897,
+          "mean_abs_state": 0.2441864162683487
+        },
+        {
+          "token": 14,
+          "top1_probability": 0.943080723285675,
+          "mean_abs_state": 0.24430227279663086
+        },
+        {
+          "token": 321,
+          "top1_probability": 0.3047252893447876,
+          "mean_abs_state": 0.24134515225887299
+        },
+        {
+          "token": 418,
+          "top1_probability": 0.24015061557292938,
+          "mean_abs_state": 0.24315044283866882
+        },
+        {
+          "token": 261,
+          "top1_probability": 0.5841822028160095,
+          "mean_abs_state": 0.2404027134180069
+        },
+        {
+          "token": 406,
+          "top1_probability": 0.3124793767929077,
+          "mean_abs_state": 0.23948220908641815
+        },
+        {
+          "token": 14,
+          "top1_probability": 0.1865627020597458,
+          "mean_abs_state": 0.241337388753891
+        },
+        {
+          "token": 321,
+          "top1_probability": 0.08900922536849976,
+          "mean_abs_state": 0.23936744034290314
+        },
+        {
+          "token": 418,
+          "top1_probability": 0.17410270869731903,
+          "mean_abs_state": 0.24108202755451202
+        },
+        {
+          "token": 261,
+          "top1_probability": 0.6804500818252563,
+          "mean_abs_state": 0.23930983245372772
+        },
+        {
+          "token": 406,
+          "top1_probability": 0.31488531827926636,
+          "mean_abs_state": 0.237319216132164
+        },
+        {
+          "token": 638,
+          "top1_probability": 0.17841003835201263,
+          "mean_abs_state": 0.2408091276884079
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.5884687900543213,
+          "mean_abs_state": 0.23970410227775574
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.43968862295150757,
+          "mean_abs_state": 0.24118487536907196
+        },
+        {
+          "token": 277,
+          "top1_probability": 0.33358219265937805,
+          "mean_abs_state": 0.24377354979515076
+        },
+        {
+          "token": 381,
+          "top1_probability": 0.25138694047927856,
+          "mean_abs_state": 0.23941843211650848
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.19209693372249603,
+          "mean_abs_state": 0.23913006484508514
+        },
+        {
+          "token": 271,
+          "top1_probability": 0.6620444655418396,
+          "mean_abs_state": 0.23728957772254944
+        },
+        {
+          "token": 335,
+          "top1_probability": 0.0351322703063488,
+          "mean_abs_state": 0.2361026555299759
+        },
+        {
+          "token": 14,
+          "top1_probability": 0.6686977744102478,
+          "mean_abs_state": 0.24020321667194366
+        },
+        {
+          "token": 348,
+          "top1_probability": 0.8959438800811768,
+          "mean_abs_state": 0.2397441864013672
+        },
+        {
+          "token": 43,
+          "top1_probability": 0.10765525698661804,
+          "mean_abs_state": 0.23796752095222473
+        },
+        {
+          "token": 305,
+          "top1_probability": 0.05576268211007118,
+          "mean_abs_state": 0.23899485170841217
+        },
+        {
+          "token": 887,
+          "top1_probability": 0.2973233163356781,
+          "mean_abs_state": 0.240958571434021
+        },
+        {
+          "token": 383,
+          "top1_probability": 0.20655134320259094,
+          "mean_abs_state": 0.2417190819978714
+        },
+        {
+          "token": 536,
+          "top1_probability": 0.15520839393138885,
+          "mean_abs_state": 0.24551555514335632
+        },
+        {
+          "token": 372,
+          "top1_probability": 0.2205699384212494,
+          "mean_abs_state": 0.24715952575206757
+        },
+        {
+          "token": 346,
+          "top1_probability": 0.22836947441101074,
+          "mean_abs_state": 0.24700763821601868
+        },
+        {
+          "token": 771,
+          "top1_probability": 0.18325209617614746,
+          "mean_abs_state": 0.24691352248191833
+        },
+        {
+          "token": 474,
+          "top1_probability": 0.10847309231758118,
+          "mean_abs_state": 0.24883104860782623
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.29328280687332153,
+          "mean_abs_state": 0.24792085587978363
+        },
+        {
+          "token": 567,
+          "top1_probability": 0.18748918175697327,
+          "mean_abs_state": 0.2532637119293213
+        },
+        {
+          "token": 71,
+          "top1_probability": 0.61652672290802,
+          "mean_abs_state": 0.2530181109905243
+        },
+        {
+          "token": 536,
+          "top1_probability": 0.28881680965423584,
+          "mean_abs_state": 0.24967096745967865
+        },
+        {
+          "token": 420,
+          "top1_probability": 0.1646156907081604,
+          "mean_abs_state": 0.24730812013149261
+        },
+        {
+          "token": 322,
+          "top1_probability": 0.10231699049472809,
+          "mean_abs_state": 0.24684029817581177
+        },
+        {
+          "token": 969,
+          "top1_probability": 0.2562040090560913,
+          "mean_abs_state": 0.24860063195228577
+        },
+        {
+          "token": 14,
+          "top1_probability": 0.22670163214206696,
+          "mean_abs_state": 0.24758069217205048
+        },
+        {
+          "token": 411,
+          "top1_probability": 0.15130679309368134,
+          "mean_abs_state": 0.24850286543369293
+        },
+        {
+          "token": 304,
+          "top1_probability": 0.20528197288513184,
+          "mean_abs_state": 0.24797432124614716
+        },
+        {
+          "token": 385,
+          "top1_probability": 0.3640832006931305,
+          "mean_abs_state": 0.24621716141700745
+        },
+        {
+          "token": 483,
+          "top1_probability": 0.11862903088331223,
+          "mean_abs_state": 0.24466590583324432
+        },
+        {
+          "token": 322,
+          "top1_probability": 0.0648268535733223,
+          "mean_abs_state": 0.24565213918685913
+        },
+        {
+          "token": 969,
+          "top1_probability": 0.10327863693237305,
+          "mean_abs_state": 0.24706204235553741
+        },
+        {
+          "token": 470,
+          "top1_probability": 0.18475353717803955,
+          "mean_abs_state": 0.24777203798294067
+        },
+        {
+          "token": 315,
+          "top1_probability": 0.1296519786119461,
+          "mean_abs_state": 0.25021296739578247
+        }
+      ],
+      "final_state_sha256": "13c081fe671d6381d2222bf3b8ac16e0514ba24bc275f488b6b217b80e9c0488",
+      "final_state_mean_abs": 0.24859312176704407
+    },
+    {
+      "prompt": "Tom and his dog",
+      "prompt_ids": [
+        1,
+        54,
+        290,
+        271,
+        341,
+        721
+      ],
+      "tokens": [
+        445,
+        381,
+        398,
+        16,
+        403,
+        445,
+        381,
+        398,
+        269,
+        545,
+        261,
+        406,
+        269,
+        322,
+        80,
+        80,
+        80,
+        16,
+        2
+      ],
+      "text": "Tom and his dog were very happy. They were very happy to have a big to bennn.",
+      "continuation": " were very happy. They were very happy to have a big to bennn.",
+      "steps": [
+        {
+          "token": 445,
+          "top1_probability": 0.23352831602096558,
+          "mean_abs_state": 0.23331324756145477
+        },
+        {
+          "token": 381,
+          "top1_probability": 0.2153073400259018,
+          "mean_abs_state": 0.24079783260822296
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.18290428817272186,
+          "mean_abs_state": 0.24464543163776398
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.7804078459739685,
+          "mean_abs_state": 0.24430245161056519
+        },
+        {
+          "token": 403,
+          "top1_probability": 0.24659904837608337,
+          "mean_abs_state": 0.2446051836013794
+        },
+        {
+          "token": 445,
+          "top1_probability": 0.3294358551502228,
+          "mean_abs_state": 0.2472667545080185
+        },
+        {
+          "token": 381,
+          "top1_probability": 0.22653907537460327,
+          "mean_abs_state": 0.2447485774755478
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.34598612785339355,
+          "mean_abs_state": 0.2425934225320816
+        },
+        {
+          "token": 269,
+          "top1_probability": 0.367300420999527,
+          "mean_abs_state": 0.24395973980426788
+        },
+        {
+          "token": 545,
+          "top1_probability": 0.12304451316595078,
+          "mean_abs_state": 0.2425643652677536
+        },
+        {
+          "token": 261,
+          "top1_probability": 0.34110602736473083,
+          "mean_abs_state": 0.24276764690876007
+        },
+        {
+          "token": 406,
+          "top1_probability": 0.15190990269184113,
+          "mean_abs_state": 0.24391822516918182
+        },
+        {
+          "token": 269,
+          "top1_probability": 0.0789346694946289,
+          "mean_abs_state": 0.24317112565040588
+        },
+        {
+          "token": 322,
+          "top1_probability": 0.052690766751766205,
+          "mean_abs_state": 0.24516509473323822
+        },
+        {
+          "token": 80,
+          "top1_probability": 0.1567470282316208,
+          "mean_abs_state": 0.2449871450662613
+        },
+        {
+          "token": 80,
+          "top1_probability": 0.41995367407798767,
+          "mean_abs_state": 0.2441568523645401
+        },
+        {
+          "token": 80,
+          "top1_probability": 0.2552717626094818,
+          "mean_abs_state": 0.24598364531993866
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.4068138003349304,
+          "mean_abs_state": 0.2503242492675781
+        },
+        {
+          "token": 2,
+          "top1_probability": 0.3975987434387207,
+          "mean_abs_state": 0.2523990869522095
+        }
+      ],
+      "final_state_sha256": "571c44260212b41761ad0f484fdcfec991505cffb16cf9852f1078a592d2cf5f",
+      "final_state_mean_abs": 0.2523990869522095
+    },
+    {
+      "prompt": "The little bird was afraid of the rain. Its mother",
+      "prompt_ids": [
+        1,
+        295,
+        364,
+        516,
+        277,
+        261,
+        72,
+        421,
+        294,
+        334,
+        264,
+        337,
+        459,
+        16,
+        443,
+        85,
+        284,
+        794
+      ],
+      "tokens": [
+        277,
+        381,
+        398,
+        271,
+        418,
+        261,
+        406,
+        645,
+        16,
+        315,
+        364,
+        413,
+        277,
+        325,
+        398,
+        16,
+        329,
+        356,
+        261,
+        406,
+        268,
+        538,
+        453,
+        389,
+        77,
+        271,
+        310,
+        380,
+        16,
+        201,
+        201,
+        295,
+        364,
+        413,
+        277,
+        381,
+        398,
+        16,
+        329,
+        356,
+        261,
+        406,
+        645,
+        71,
+        340,
+        16,
+        329,
+        277,
+        325,
+        398,
+        353,
+        321,
+        356,
+        261,
+        406,
+        645,
+        71,
+        340,
+        16,
+        329
+      ],
+      "text": "The little bird was afraid of the rain. Its mother was very happy and saw a big hug. The little girl was so happy. She had a big bagicalk and her mom.\n\nThe little girl was very happy. She had a big hugest. She was so happy that she had a big hugest. She",
+      "continuation": " was very happy and saw a big hug. The little girl was so happy. She had a big bagicalk and her mom.\n\nThe little girl was very happy. She had a big hugest. She was so happy that she had a big hugest. She",
+      "steps": [
+        {
+          "token": 277,
+          "top1_probability": 0.07479799538850784,
+          "mean_abs_state": 0.23606424033641815
+        },
+        {
+          "token": 381,
+          "top1_probability": 0.2318151295185089,
+          "mean_abs_state": 0.23601475358009338
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.051157474517822266,
+          "mean_abs_state": 0.2380186915397644
+        },
+        {
+          "token": 271,
+          "top1_probability": 0.4954601526260376,
+          "mean_abs_state": 0.23955120146274567
+        },
+        {
+          "token": 418,
+          "top1_probability": 0.049759287387132645,
+          "mean_abs_state": 0.23933415114879608
+        },
+        {
+          "token": 261,
+          "top1_probability": 0.21216799318790436,
+          "mean_abs_state": 0.23974043130874634
+        },
+        {
+          "token": 406,
+          "top1_probability": 0.2777990400791168,
+          "mean_abs_state": 0.2402232438325882
+        },
+        {
+          "token": 645,
+          "top1_probability": 0.06080712005496025,
+          "mean_abs_state": 0.2366064190864563
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.9221192002296448,
+          "mean_abs_state": 0.2386849969625473
+        },
+        {
+          "token": 315,
+          "top1_probability": 0.260009765625,
+          "mean_abs_state": 0.24082832038402557
+        },
+        {
+          "token": 364,
+          "top1_probability": 0.11506637930870056,
+          "mean_abs_state": 0.24146826565265656
+        },
+        {
+          "token": 413,
+          "top1_probability": 0.6310064792633057,
+          "mean_abs_state": 0.2397887259721756
+        },
+        {
+          "token": 277,
+          "top1_probability": 0.234466090798378,
+          "mean_abs_state": 0.23675085604190826
+        },
+        {
+          "token": 325,
+          "top1_probability": 0.35203421115875244,
+          "mean_abs_state": 0.23671753704547882
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.3745185136795044,
+          "mean_abs_state": 0.2355474978685379
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.3074924647808075,
+          "mean_abs_state": 0.23643888533115387
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.2839523255825043,
+          "mean_abs_state": 0.23618222773075104
+        },
+        {
+          "token": 356,
+          "top1_probability": 0.08014149963855743,
+          "mean_abs_state": 0.23787446320056915
+        },
+        {
+          "token": 261,
+          "top1_probability": 0.3320424556732178,
+          "mean_abs_state": 0.23796045780181885
+        },
+        {
+          "token": 406,
+          "top1_probability": 0.25125259160995483,
+          "mean_abs_state": 0.23734146356582642
+        },
+        {
+          "token": 268,
+          "top1_probability": 0.0636390820145607,
+          "mean_abs_state": 0.23741762340068817
+        },
+        {
+          "token": 538,
+          "top1_probability": 0.21194598078727722,
+          "mean_abs_state": 0.2352684587240219
+        },
+        {
+          "token": 453,
+          "top1_probability": 0.13777169585227966,
+          "mean_abs_state": 0.23696021735668182
+        },
+        {
+          "token": 389,
+          "top1_probability": 0.43053922057151794,
+          "mean_abs_state": 0.23880206048488617
+        },
+        {
+          "token": 77,
+          "top1_probability": 0.24204905331134796,
+          "mean_abs_state": 0.23708176612854004
+        },
+        {
+          "token": 271,
+          "top1_probability": 0.18610747158527374,
+          "mean_abs_state": 0.24004849791526794
+        },
+        {
+          "token": 310,
+          "top1_probability": 0.20035195350646973,
+          "mean_abs_state": 0.2399768978357315
+        },
+        {
+          "token": 380,
+          "top1_probability": 0.15768368542194366,
+          "mean_abs_state": 0.23940463364124298
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.13256524503231049,
+          "mean_abs_state": 0.23910191655158997
+        },
+        {
+          "token": 201,
+          "top1_probability": 0.23595978319644928,
+          "mean_abs_state": 0.24282129108905792
+        },
+        {
+          "token": 201,
+          "top1_probability": 0.9938191771507263,
+          "mean_abs_state": 0.24466349184513092
+        },
+        {
+          "token": 295,
+          "top1_probability": 0.17668399214744568,
+          "mean_abs_state": 0.24673636257648468
+        },
+        {
+          "token": 364,
+          "top1_probability": 0.33029958605766296,
+          "mean_abs_state": 0.24477310478687286
+        },
+        {
+          "token": 413,
+          "top1_probability": 0.9418007731437683,
+          "mean_abs_state": 0.24426759779453278
+        },
+        {
+          "token": 277,
+          "top1_probability": 0.24806471168994904,
+          "mean_abs_state": 0.24204768240451813
+        },
+        {
+          "token": 381,
+          "top1_probability": 0.28256601095199585,
+          "mean_abs_state": 0.24498136341571808
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.40280795097351074,
+          "mean_abs_state": 0.24408169090747833
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.3394959270954132,
+          "mean_abs_state": 0.24055686593055725
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.6678237318992615,
+          "mean_abs_state": 0.2407580018043518
+        },
+        {
+          "token": 356,
+          "top1_probability": 0.09095501899719238,
+          "mean_abs_state": 0.24119128286838531
+        },
+        {
+          "token": 261,
+          "top1_probability": 0.257854700088501,
+          "mean_abs_state": 0.24170278012752533
+        },
+        {
+          "token": 406,
+          "top1_probability": 0.16331659257411957,
+          "mean_abs_state": 0.23934176564216614
+        },
+        {
+          "token": 645,
+          "top1_probability": 0.08178900927305222,
+          "mean_abs_state": 0.23926077783107758
+        },
+        {
+          "token": 71,
+          "top1_probability": 0.2544991970062256,
+          "mean_abs_state": 0.23774895071983337
+        },
+        {
+          "token": 340,
+          "top1_probability": 0.0779440626502037,
+          "mean_abs_state": 0.23881997168064117
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.11029413342475891,
+          "mean_abs_state": 0.23900508880615234
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.6655603647232056,
+          "mean_abs_state": 0.2403733730316162
+        },
+        {
+          "token": 277,
+          "top1_probability": 0.08762046694755554,
+          "mean_abs_state": 0.24371850490570068
+        },
+        {
+          "token": 325,
+          "top1_probability": 0.22797028720378876,
+          "mean_abs_state": 0.2422819435596466
+        },
+        {
+          "token": 398,
+          "top1_probability": 0.4236501157283783,
+          "mean_abs_state": 0.2387838363647461
+        },
+        {
+          "token": 353,
+          "top1_probability": 0.40683120489120483,
+          "mean_abs_state": 0.23900966346263885
+        },
+        {
+          "token": 321,
+          "top1_probability": 0.6709152460098267,
+          "mean_abs_state": 0.24038520455360413
+        },
+        {
+          "token": 356,
+          "top1_probability": 0.21871809661388397,
+          "mean_abs_state": 0.24375919997692108
+        },
+        {
+          "token": 261,
+          "top1_probability": 0.15725688636302948,
+          "mean_abs_state": 0.2453835904598236
+        },
+        {
+          "token": 406,
+          "top1_probability": 0.11761261522769928,
+          "mean_abs_state": 0.2423761636018753
+        },
+        {
+          "token": 645,
+          "top1_probability": 0.09153948724269867,
+          "mean_abs_state": 0.24255241453647614
+        },
+        {
+          "token": 71,
+          "top1_probability": 0.22967354953289032,
+          "mean_abs_state": 0.2414238154888153
+        },
+        {
+          "token": 340,
+          "top1_probability": 0.14407241344451904,
+          "mean_abs_state": 0.2435961216688156
+        },
+        {
+          "token": 16,
+          "top1_probability": 0.23882780969142914,
+          "mean_abs_state": 0.24225760996341705
+        },
+        {
+          "token": 329,
+          "top1_probability": 0.5803385972976685,
+          "mean_abs_state": 0.24394433200359344
+        }
+      ],
+      "final_state_sha256": "0b2a6a60a7f5640d98b6bf1554424f57801c49e8ffd0c0029b21263ea447634f",
+      "final_state_mean_abs": 0.2468549609184265
+    }
+  ]
+}

web/rec_gain.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29250b4f7d8f2d036ec7e95207dcc67384868886cafc62a9d9f5c116e1bfa594
+size 197572

web/wte.f32

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2ce3cc6a0d92633b2c0de0e22d8fc2545369c0f00bab28b6366f698d2458ba8
+size 131072