---
library_name: pytorch
license: apache-2.0
datasets:
  - roneneldan/TinyStories
  - Salesforce/wikitext
  - HuggingFaceTB/everyday-conversations-llama3.1-2k
tags:
  - atomslm
  - language-model
  - shared-weights
  - pytorch
---

# AtomSLM-1.2M

**AtomSLM** is a compact, shared-weight language model family built on the 
AtomNet architecture (Shared Weight Core + Per-Layer FiLM Modulation). 
One shared weight core is reused across all N layers, with tiny per-layer 
FiLM vectors providing the only per-layer state — deep reasoning at minimal parameter cost.

## Model Details

| Field            | Value                          |
|------------------|--------------------------------|
| Architecture     | AtomNet (shared-core + FiLM)   |
| Parameters       | 1.094M                      |
| Vocab size       | 4096                        |
| d_model           | 192                          |
| Layers           | 10                          |
| FFN multiplier   | 2.0                       |
| Context window   | 256 tokens                |
| Weight tying     | True                     |
| Best val loss    | 3.1145                      |
| Best PPL (val)   | 22.52                        |
| Trained steps    | 5000                          |

## Training

Trained on the following datasets with a custom BPE tokenizer (vocab size matching the config above):

- `roneneldan/TinyStories`
- `wikitext-2-raw-v1`
- `wikitext-103-raw-v1`
- `hand-crafted-conversations`
- `HuggingFaceTB/everyday-conversations-llama3.1-2k`

### Hyperparameters

```json
{
  "data_dir": "data/processed",
  "save_dir": "runs/AtomSLM-1.2M",
  "config": "AtomSLM-1.2M",
  "steps": 5000,
  "eval_every": 100,
  "save_every": 500,
  "batch_size": 32,
  "seq_len": 256,
  "lr": 0.0005,
  "lr_min": 5e-05,
  "warmup": 1000,
  "grad_clip": 1.0,
  "dropout": 0.1,
  "device": "auto",
  "resume": null,
  "compile": false,
  "amp": false,
  "core_warmup_steps": 0
}
```

## Training Dashboard

![Training Dashboard](dashboard.png)

## Benchmark vs Reference Models

![Comparison Charts](compare_charts.png)

## Usage

```python
import torch
from models.atomgpt import AtomSLM

ckpt  = torch.load('pytorch_model.bin', map_location='cpu')
model = AtomSLM(ckpt['config'])
model.load_state_dict(ckpt['model_state'])
model.eval()
```

## License

Apache 2.0