redredredredredred commited on
Commit
cdffd17
·
verified ·
1 Parent(s): 7eeb122

Link to Phase 3 scaling-law checkpoints repo

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -50,7 +50,7 @@ This is Phase 1 of a larger build documented at [github.com/Ishaanred/slm-from-s
50
 
51
  This model is undertrained on purpose. At ~80M tokens it has seen roughly 5% of what a 77M model needs to converge (Chinchilla suggests ~1.5B tokens for this size). The output is grammatical English with plausible local structure but no coherence across sentences.
52
 
53
- A companion 50M model trained on the same 80M tokens reached a lower validation loss (5.12). At a fixed, small token budget the smaller model wins, because the larger one is more undertrained relative to its capacity. That gap is expected to reverse with longer training. Phase 3 of the project runs the controlled scaling-law experiments that test this.
54
 
55
  ### Sample output
56
 
 
50
 
51
  This model is undertrained on purpose. At ~80M tokens it has seen roughly 5% of what a 77M model needs to converge (Chinchilla suggests ~1.5B tokens for this size). The output is grammatical English with plausible local structure but no coherence across sentences.
52
 
53
+ A companion 50M model trained on the same 80M tokens reached a lower validation loss (5.12). At a fixed, small token budget the smaller model wins, because the larger one is more undertrained relative to its capacity. That gap is expected to reverse with longer training. Phase 3 of the project runs the controlled scaling-law experiments that test this. See [slm-from-scratch-phase3-scaling](https://huggingface.co/redredredredredred/slm-from-scratch-phase3-scaling) for those checkpoints and the scaling-law results.
54
 
55
  ### Sample output
56