PyTorch
atomslm
language-model
shared-weights
Sqersters commited on
Commit
a3484dd
·
verified ·
1 Parent(s): e3d130b

Upload AtomSLM checkpoint

Browse files
Files changed (4) hide show
  1. .gitattributes +2 -0
  2. README.md +37 -15
  3. compare_charts.png +3 -0
  4. dashboard.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ compare_charts.png filter=lfs diff=lfs merge=lfs -text
37
+ dashboard.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,34 +1,48 @@
1
  ---
2
  library_name: pytorch
 
 
 
 
3
  tags:
4
  - atomslm
5
  - language-model
6
  - shared-weights
 
7
  ---
8
 
9
  # AtomSLM-1.2M
10
 
11
- **AtomSLM** is a compact, shared-weight language model family built on the
12
- AtomNet architecture (Shared Weight Core + Per-Layer FiLM Modulation).
 
 
13
 
14
  ## Model Details
15
 
16
- | Field | Value |
17
- |---------------|---------------------------|
18
- | Architecture | AtomNet (shared-core FiLM)|
19
- | Parameters | 1.094M |
20
- | Vocab size | 4096 |
21
- | d_model | 192 |
22
- | Layers | 10 |
23
- | FFN mult | 2.0 |
24
- | Max seq len | 256 |
25
- | Weight tying | True |
26
- | Val loss | 3.1145 |
27
- | Trained steps | 5000 |
 
28
 
29
  ## Training
30
 
31
- Trained on **TinyStories + WikiText-2** with a custom BPE tokenizer.
 
 
 
 
 
 
32
 
33
  ### Hyperparameters
34
 
@@ -55,6 +69,14 @@ Trained on **TinyStories + WikiText-2** with a custom BPE tokenizer.
55
  }
56
  ```
57
 
 
 
 
 
 
 
 
 
58
  ## Usage
59
 
60
  ```python
 
1
  ---
2
  library_name: pytorch
3
+ datasets:
4
+ - roneneldan/TinyStories
5
+ - Salesforce/wikitext
6
+ - HuggingFaceTB/everyday-conversations-llama3.1-2k
7
  tags:
8
  - atomslm
9
  - language-model
10
  - shared-weights
11
+ - pytorch
12
  ---
13
 
14
  # AtomSLM-1.2M
15
 
16
+ **AtomSLM** is a compact, shared-weight language model family built on the
17
+ AtomNet architecture (Shared Weight Core + Per-Layer FiLM Modulation).
18
+ One shared weight core is reused across all N layers, with tiny per-layer
19
+ FiLM vectors providing the only per-layer state — deep reasoning at minimal parameter cost.
20
 
21
  ## Model Details
22
 
23
+ | Field | Value |
24
+ |------------------|--------------------------------|
25
+ | Architecture | AtomNet (shared-core + FiLM) |
26
+ | Parameters | 1.094M |
27
+ | Vocab size | 4096 |
28
+ | d\_model | 192 |
29
+ | Layers | 10 |
30
+ | FFN multiplier | 2.0 |
31
+ | Context window | 256 tokens |
32
+ | Weight tying | True |
33
+ | Best val loss | 3.1145 |
34
+ | Best PPL (val) | 22.52 |
35
+ | Trained steps | 5000 |
36
 
37
  ## Training
38
 
39
+ Trained on the following datasets with a custom BPE tokenizer (vocab size matching the config above):
40
+
41
+ - `roneneldan/TinyStories`
42
+ - `wikitext-2-raw-v1`
43
+ - `wikitext-103-raw-v1`
44
+ - `hand-crafted-conversations`
45
+ - `HuggingFaceTB/everyday-conversations-llama3.1-2k`
46
 
47
  ### Hyperparameters
48
 
 
69
  }
70
  ```
71
 
72
+ ## Training Dashboard
73
+
74
+ ![Training Dashboard](dashboard.png)
75
+
76
+ ## Benchmark vs Reference Models
77
+
78
+ ![Comparison Charts](compare_charts.png)
79
+
80
  ## Usage
81
 
82
  ```python
compare_charts.png ADDED

Git LFS Details

  • SHA256: cfe5cd4e4b02e708fed81ed5721bf8dca8136cacccccc0f71331c2943942a6c7
  • Pointer size: 131 Bytes
  • Size of remote file: 140 kB
dashboard.png ADDED

Git LFS Details

  • SHA256: 0a6fa942b8e212fa16b0276269add2c231931b79e05eb0a32678520e918de3c0
  • Pointer size: 131 Bytes
  • Size of remote file: 125 kB