Fly Wordbrain β rank 32
A 2,343,125-parameter language model on a frozen fruit-fly connectome β 22.5Γ smaller than the model it derives from, and still ahead of it on held-out text.
This is the smaller sibling of fly-wordbrain-rank64, which is the recommended baseline. Everything about the architecture is identical except the readout rank: 49,393 β 32 β 1,024 instead of β 64 β.
Research and receipts: fly-wordbrain
What halving the readout again costs
200 fresh TinyStories, 45,059 next-token targets, selecting no checkpoint.
| Model | Parameters | Audit CE | Audit top-1 | ΞCE vs reference (95% CI) |
|---|---|---|---|---|
| ngxson reference | 52,756,661 | 3.9882 | 31.38% | β |
| rank 64 | 3,956,469 | 3.2802 | 33.37% | β0.708 [β0.738, β0.679] |
| rank 32 (this) | 2,343,125 | 3.3091 | 32.58% | β0.679 [β0.711, β0.647] |
Relative to rank 64: 1,613,344 fewer parameters for 0.029 nats and 0.79 accuracy points. On the selection-validation split that gap reads as 1.42 points, and its generated text has more visible failures on generic prompts. Take this one when size matters more than the last point of quality.
| Component | Parameters |
|---|---|
| Encoder (embedding + 8 input projections) | 482,816 |
| Neuron gain, recurrent gain, bias | 148,179 |
| Output LayerNorm | 98,786 |
| Factorized readout (rank 32) | 1,613,344 |
| Total | 2,343,125 |
No synaptic weight was modified. The frozen-buffer digests in manifest.json are
byte-identical to the reference model's and to rank 64's β no edge changed, none rewired.
The 49,393 neurons' own dynamics are trained, though: 148,179 learned gain / recurrent
gain / bias values, part of the reference architecture rather than an addition here.
Against the seed-rebuilt initialization, gain moves 60.57% in relative L2, rec_gain
13.04%, and the effective per-neuron scaling gain Γ rec_gain 56.18%. rec_gain
rescales a neuron's whole incoming sum by one factor, so it cannot change the relative
strengths or signs of individual synapses.
Both retained selectors land on update 16,600 here, so min-ce.safetensors and
max-accuracy.safetensors hold the same weights under two different selection receipts.
Caveats
The comparison against the released reference is not controlled β its training stories and trainer are unknown. The controlled finding is that constraining the readout rank acts as a regularizer against overfitting 1,000 short stories; it says nothing about whether the fly's wiring is a good prior for language. A randomised-graph control has since measured that directly: rewiring the connectome at random, with every degree and weight preserved, costs 0.0100 nats [+0.0026, +0.0175] β about 1% of this architecture's margin over the reference. The wiring contributes something, and very little of it.
The connectome is not duplicated here; it lives in fly-connectome-49k, with the complete 166,700-neuron graph at fly-connectome-malecns-166k. Loading instructions and the full step function are on the rank-64 card.
Licence
Weights CC BY 4.0 β derived from MaleCNS v1.0 (FlyEM / HHMI Janelia, University of
Cambridge, MRC LMB, Google Research) and the
ngxson/fly-llm-hf architecture. Code MIT.
TinyStories is not redistributed.