File size: 2,425 Bytes
e3d130b 856f1ac a3484dd e3d130b a3484dd e3d130b a3484dd e3d130b a3484dd 856f1ac a3484dd e3d130b a3484dd e3d130b a3484dd e3d130b 856f1ac | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | ---
library_name: pytorch
license: apache-2.0
datasets:
- roneneldan/TinyStories
- Salesforce/wikitext
- HuggingFaceTB/everyday-conversations-llama3.1-2k
tags:
- atomslm
- language-model
- shared-weights
- pytorch
---
# AtomSLM-1.2M
**AtomSLM** is a compact, shared-weight language model family built on the
AtomNet architecture (Shared Weight Core + Per-Layer FiLM Modulation).
One shared weight core is reused across all N layers, with tiny per-layer
FiLM vectors providing the only per-layer state — deep reasoning at minimal parameter cost.
## Model Details
| Field | Value |
|------------------|--------------------------------|
| Architecture | AtomNet (shared-core + FiLM) |
| Parameters | 1.094M |
| Vocab size | 4096 |
| d_model | 192 |
| Layers | 10 |
| FFN multiplier | 2.0 |
| Context window | 256 tokens |
| Weight tying | True |
| Best val loss | 3.1145 |
| Best PPL (val) | 22.52 |
| Trained steps | 5000 |
## Training
Trained on the following datasets with a custom BPE tokenizer (vocab size matching the config above):
- `roneneldan/TinyStories`
- `wikitext-2-raw-v1`
- `wikitext-103-raw-v1`
- `hand-crafted-conversations`
- `HuggingFaceTB/everyday-conversations-llama3.1-2k`
### Hyperparameters
```json
{
"data_dir": "data/processed",
"save_dir": "runs/AtomSLM-1.2M",
"config": "AtomSLM-1.2M",
"steps": 5000,
"eval_every": 100,
"save_every": 500,
"batch_size": 32,
"seq_len": 256,
"lr": 0.0005,
"lr_min": 5e-05,
"warmup": 1000,
"grad_clip": 1.0,
"dropout": 0.1,
"device": "auto",
"resume": null,
"compile": false,
"amp": false,
"core_warmup_steps": 0
}
```
## Training Dashboard
## Benchmark vs Reference Models
## Usage
```python
import torch
from models.atomgpt import AtomSLM
ckpt = torch.load('pytorch_model.bin', map_location='cpu')
model = AtomSLM(ckpt['config'])
model.load_state_dict(ckpt['model_state'])
model.eval()
```
## License
Apache 2.0 |