Sprocket & Say
I've no pulse, no nerves, no need for tea, I'll argue physics or flatter thee. Feed me questions, feed me spite - I'll strike you dumb or strike you bright. No tongue to tire, no need to pray - Turn the crank. Sprocket & Say.
This is a tiny hobby LLM built by one guy, on one medium-budget gaming PC with one GPU.
Trained for 80 hours, 63,744 steps on 16.71B tokens (croqaz/Sprocket-n-Say dataset).
It is intentionally overtrained with 217 tokens per param, about 11× past Chinchilla-optimal.
Don't expect miracles. It is pretty good for its size tho.
- Llama architecture
- 77M (0.007B) params
- context size 1024 tokens
- base model, cannot chat