Question
Do you count n-gram parameters as model parameters? Because n-gram can be disk streamed.
Just to keep things simple im gonna go with yes
Do you count n-gram parameters as model parameters? Because n-gram can be disk streamed.
Give up on Engram; that's my componet and you stole it.
P.S Nah just kidding, but lwk I hoped that I was the only one to use them.
Give up on Engram; that's my componet and you stole it.
P.S Nah just kidding, but lwk I hoped that I was the only one to use them.
OpenCerebral has adopted them for Boris-1.7. and i'm doing some test runs for my future models with ngram too lol
yeah they seem very very promising
Give up on Engram; that's my componet and you stole it.
P.S Nah just kidding, but lwk I hoped that I was the only one to use them.
OpenCerebral has adopted them for Boris-1.7. and i'm doing some test runs for my future models with ngram too lol
Yeah they might be the new meta.
Based on my ablations, it gives +1 points on arc_easy at 3M scale.
At parameter matched?
At parameter matched?
Yep, in fact, the Engram one had 40k less params then the baseline.
At parameter matched?
And as for Boris, at least — it beat my original Boris-125M with only 60M dense params and 30M n-gram. Granted, it saw a different data source and a custom tokenizer, but still.
If you look at the benchmark image, you can see it with and without the n-gram, and the n-gram helps it very much on ARC.
My ablation just finished I'll give results