README.md

metadata

library_name: pytorch
license: apache-2.0
datasets:
  - roneneldan/TinyStories
  - Salesforce/wikitext
  - HuggingFaceTB/everyday-conversations-llama3.1-2k
tags:
  - atomslm
  - language-model
  - shared-weights
  - pytorch

AtomSLM-1.2M

AtomSLM is a compact, shared-weight language model family built on the AtomNet architecture (Shared Weight Core + Per-Layer FiLM Modulation). One shared weight core is reused across all N layers, with tiny per-layer FiLM vectors providing the only per-layer state — deep reasoning at minimal parameter cost.

Model Details

Field	Value
Architecture	AtomNet (shared-core + FiLM)
Parameters	1.094M
Vocab size	4096
d_model	192
Layers	10
FFN multiplier	2.0
Context window	256 tokens
Weight tying	True
Best val loss	3.1145
Best PPL (val)	22.52
Trained steps	5000

Training

Trained on the following datasets with a custom BPE tokenizer (vocab size matching the config above):

• roneneldan/TinyStories
• wikitext-2-raw-v1
• wikitext-103-raw-v1
• hand-crafted-conversations
• HuggingFaceTB/everyday-conversations-llama3.1-2k

Hyperparameters

{
  "data_dir": "data/processed",
  "save_dir": "runs/AtomSLM-1.2M",
  "config": "AtomSLM-1.2M",
  "steps": 5000,
  "eval_every": 100,
  "save_every": 500,
  "batch_size": 32,
  "seq_len": 256,
  "lr": 0.0005,
  "lr_min": 5e-05,
  "warmup": 1000,
  "grad_clip": 1.0,
  "dropout": 0.1,
  "device": "auto",
  "resume": null,
  "compile": false,
  "amp": false,
  "core_warmup_steps": 0
}

Training Dashboard

Benchmark vs Reference Models

Usage

import torch
from models.atomgpt import AtomSLM

ckpt  = torch.load('pytorch_model.bin', map_location='cpu')
model = AtomSLM(ckpt['config'])
model.load_state_dict(ckpt['model_state'])
model.eval()

License

Apache 2.0