File size: 3,120 Bytes
45e7712 53d5954 45e7712 53d5954 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
license: cc0-1.0
language:
- en
---
# Atto: Extreme Intelligence Density Research
Atto is an exploration into the fundamental limits of **Intelligence Density** — how much knowledge and generative capability can be packed into a neural network with a strictly limited parameter budget.
This project focuses on the "sub-kiloparameter" and "low-kiloparameter" regime, training models to generate Shakespearean text with as few as 64 parameters.
## The Atto Series
| Model | Parameters | Context | Weights Size (JSON) | Val Loss |
| :--- | :--- | :--- | :--- | :--- |
| **atto-64** | 64 | 3 | 1.8 KB | 2.59 |
| **atto-128** | 128 | 7 | 3.5 KB | 2.83 |
| **atto-256** | 256 | 8 | 6.0 KB | 2.33 |
| **atto-512** | 512 | 16 | 11.8 KB | 2.44 |
| **atto-1024** | 1,024 | 8 | 22.3 KB | 2.11 |
| **atto-2048** | 2,048 | 24 | 44.3 KB | 2.15 |
| **atto-4096** | 4,096 | 56 | 86.4 KB | 2.40 |
| **atto-8192** | 8,192 | 28 | 172.7 KB | 1.91 |
| **atto-16384** | 16,384 | 60 | ~640 KB | 2.11 |
## Research Findings: Intelligence Density
1. **Architecture Matters**: At the sub-1000 parameter scale, standard Transformers are highly inefficient due to the overhead of Attention and LayerNorm. Our custom **Neural N-Gram (AttoLM)** architecture ensures that every single parameter directly participates in character prediction.
2. **The Embedding Threshold**: We found that moving from 8-dimensional to 16-dimensional embeddings (at 8,192 parameters) creates a significant jump in coherence, allowing the model to represent complex character relationships.
3. **Context vs. Width**: In extremely small models, there is a sharp trade-off between the context window (memory) and embedding dimensionality (representation). Our 8,192 and 16,384 models prioritize a balance that favors realistic word formation.
## Next Steps
This is just a **first step** in making intelligence very dense. By optimizing weight initialization, custom activation functions, and even more extreme parameter-tying, we believe it is possible to achieve "readable Shakespeare" with even fewer than 1,000 parameters.
## Usage
### Training
To train the base series, run:
```bash
python3 train_atto.py
```
### Sampling
To evaluate all trained models:
```bash
python3 sample.py
```
The models are exported as dependency-free JSON files in the `models/` directory, ready for client-side inference in a web browser.
### Sample generations:
```
============================================================
atto-8192 | 8192 params | embd=16 ctx=28 vocab=64
============================================================
prompt="the":
Math Laer axfourith tipht's gord me hour hace (remaat ond,
I'll wore ser ar now pre's for word to styous the mall, stpoul folthis yow apt and be a
prompt="to be":
CPon. How gue. O- whut feathent. Thou the in ap bast. gos A thing of be rith nosset?
[Tiths that hintend kyele in younk hore;
Gat sgees wis
prompt="Ham":
. HaCleata,
Wlotsef yow preerant fore thipe matte of iche in you?
And spour, the tang offe herees welr then[foritr her veut arve id for houn w
``` |