Upload AtomSLM checkpoint

README.md

@@ -8,31 +8,23 @@ datasets:
 tags:
   - atomslm
   - language-model
-  - shared-weights
   - pytorch
 ---
 
 # AtomSLM-600K
 
-**AtomSLM** is a compact, shared-weight language model family built on the 
-AtomNet architecture (Shared Weight Core + Per-Layer FiLM Modulation). 
-One shared weight core is reused across all N layers, with tiny per-layer 
-FiLM vectors providing the only per-layer state — deep reasoning at minimal parameter cost.
+**AtomSLM** is a compact language model family achieving competitive 
+performance at minimal parameter cost through proprietary architecture.
 
 ## Model Details
 
 | Field            | Value                          |
 |------------------|--------------------------------|
-| Architecture     | AtomNet (shared-core + FiLM)   |
 | Parameters       | 0.613M                      |
 | Vocab size       | 4096                        |
-| d_model           | 120                          |
-| Layers           | 8                          |
-| FFN multiplier   | 2.0                       |
 | Context window   | 256 tokens                |
-| Weight tying     | True                     |
-| Best val loss    | 3.3455                      |
-| Best PPL (val)   | 28.37                        |
+| Best val loss    | 3.3553                      |
+| Best PPL (val)   | 28.65                        |
 | Trained steps    | 4900                          |
 
 ## Training
@@ -66,7 +58,8 @@ Trained on the following datasets with a custom BPE tokenizer (vocab size matchi
   "resume": null,
   "compile": false,
   "amp": false,
-  "core_warmup_steps": 0
+  "core_warmup_steps": 0,
+  "gated": false
 }
 ```
 
@@ -82,11 +75,17 @@ Trained on the following datasets with a custom BPE tokenizer (vocab size matchi
 
 ```python
 import torch
+from safetensors.torch import load_file
 from models.atomgpt import AtomSLM
+from models.config import AtomSLMConfig
 
-ckpt  = torch.load('pytorch_model.bin', map_location='cpu')
-model = AtomSLM(ckpt['config'])
-model.load_state_dict(ckpt['model_state'])
+# Load config and weights
+import json
+cfg = AtomSLMConfig(**json.load(open('config.json')))
+state_dict = load_file('model.safetensors')
+
+model = AtomSLM(cfg)
+model.load_state_dict(state_dict)
 model.eval()
 ```
 

config.json

@@ -1,12 +1,7 @@
 {
   "vocab_size": 4096,
-  "d_model": 120,
-  "n_layers": 8,
-  "ffn_mult": 2.0,
-  "dropout": 0.1,
   "max_seq_len": 256,
-  "tie_weights": true,
   "architecture": "AtomSLM",
-  "val_loss": 3.345488176345825,
+  "val_loss": 3.355283432006836,
   "step": 4900
 }

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac3d922bcecc97b41e88f9282999abd99f0a59d856f42fb9bd95fd607413719a
+size 4422760