Link to Phase 3 scaling-law checkpoints repo
Browse files
README.md
CHANGED
|
@@ -50,7 +50,7 @@ This is Phase 1 of a larger build documented at [github.com/Ishaanred/slm-from-s
|
|
| 50 |
|
| 51 |
This model is undertrained on purpose. At ~80M tokens it has seen roughly 5% of what a 77M model needs to converge (Chinchilla suggests ~1.5B tokens for this size). The output is grammatical English with plausible local structure but no coherence across sentences.
|
| 52 |
|
| 53 |
-
A companion 50M model trained on the same 80M tokens reached a lower validation loss (5.12). At a fixed, small token budget the smaller model wins, because the larger one is more undertrained relative to its capacity. That gap is expected to reverse with longer training. Phase 3 of the project runs the controlled scaling-law experiments that test this.
|
| 54 |
|
| 55 |
### Sample output
|
| 56 |
|
|
|
|
| 50 |
|
| 51 |
This model is undertrained on purpose. At ~80M tokens it has seen roughly 5% of what a 77M model needs to converge (Chinchilla suggests ~1.5B tokens for this size). The output is grammatical English with plausible local structure but no coherence across sentences.
|
| 52 |
|
| 53 |
+
A companion 50M model trained on the same 80M tokens reached a lower validation loss (5.12). At a fixed, small token budget the smaller model wins, because the larger one is more undertrained relative to its capacity. That gap is expected to reverse with longer training. Phase 3 of the project runs the controlled scaling-law experiments that test this. See [slm-from-scratch-phase3-scaling](https://huggingface.co/redredredredredred/slm-from-scratch-phase3-scaling) for those checkpoints and the scaling-law results.
|
| 54 |
|
| 55 |
### Sample output
|
| 56 |
|