Zero vocabulary tax, character-level sequence expansion, and immaculate Roblox priors in 10.7M architectures
Hi Brody,
"Times it brought up Roblox unprompted: 2" and "Vibes: Immaculate" is easily the most honest benchmark suite on Hugging Face this month. Training a pure 10.7M model to exclusively simulate the linguistic entropy of YouTube Shorts comments is fantastic chaotic engineering.
Looking at the underlying mechanics of a 6-layer, 384-dim character-level configuration:
The zero vocabulary tax advantage:
While modern sub-50M models often waste 20% to 35% of their parameter budget on massive 32k or 49k BPE embedding lookup tables, a character-level model has a microscopic vocabulary. Virtually 100% of your 10.7M parameters are allocated directly into active math and attention layers.The character-level sequence inflation tax:
The trade-off with character-level causal GPTs is token fertility: spelling out typical Shorts chaos ("AHHH MOST POPULAR SHORT OF APPIL?????????!") eats 50+ tokens in seconds.
Because standard quadratic attention scales as O(L^2), parsing or continuing across longer comment threads quickly saturates the attention window, while single characters carry very low individual semantic salience.Constant O(1) state updates for character streams:
In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore sub-quadratic scaling using a 3:1 macro-topology (DGDA linear recurrence paired with sparse latent attention):
Character-level modeling is historically the natural habitat for recurrent state transitions. Instead of storing an expanding quadratic attention cache for every single letter, space, and emoji, linear recurrence continuously folds incoming characters into an associative memory matrix in constant O(1) space.
This allows multi-thousand character rollouts with strictly flat decode latency and near-zero memory footprint.
Did you train with a fixed byte-level vocabulary (e.g. 256 bytes) to capture any raw unicode/emoji directly, or did you prune to a custom character set?
Best,
Andrew
Hello Andrew.
Thanks for reaching out to me!
I use a custom charset I believe.
If you would like I could give you access to one of my models in beta right now!
Also I will make sure I can try Maba later. Thanks for reaching out!
Best,
Brody.
Hello Andrew.
I have got the chat running.
Your script loads the checkpoint clean. There are no missing weights which is good. But your output of the AI is kinda weird.
Your AI is outputting word salad. I would like to check with you 2 things.
- Which tokenizer did you use? I assumed GPT 2 but it must be a different one.
- Is the maba-1.5-130m completely trained?
I am assuming it must be number 1. If you have answers. please reach out.
Best,
Brody.
Hi Brody,
Thanks for testing it out! Would love to check out your beta model :)
You actually guessed both right:
- Tokenizer: Maba uses a 32,768 vocab (32k byte-level BPE), not GPT-2 (50,257). Feeding GPT-2 token IDs into a 32k embedding table completely scrambles the weights into word salad.
- Checkpoint: The 1.5 checkpoint was just an experimental prototype to test the architecture, not a fully converged dialogue model.
Right now I'm training maba-v2-5m, which is built specifically for real multi-turn chat.
How can I get access to your beta? Super curious to try it!
Best,
Andrew
Hello Andrew,
I will put my models stuff on a private / unlisted space.
I made a custom GPT that can actually chat!
Anyways, I will give you the link as soon as I can. Thanks!
Best,
Brody
Hi Brody,
That sounds awesome! Really looking forward to checking out the Space and seeing how your custom GPT chats. Take your time setting it up, just drop the link whenever you're ready :)
Best,
Andrew
Hello Andrew,
The link is ready! You just gotta accept access to my org. https://huggingface.co/BRODILOFT/wondererbeta
Best, Brody.
Also the AI is non commerical due to the datasets I used.
Best,
Brody.
Hi Brody,
Joined the org and checked out the repo!
Really smart choice keeping the vocabulary at 4,096. On an 8.8M model that leaves almost the entire parameter budget for the actual transformer layers instead of wasting it on a massive lookup table. Also, huge respect for the benchmarks section in the card - seeing raw numbers and funny outputs like Liberia for France's capital is genuine open-science transparency.
Totally noted on the non-commercial datasets (Alpaca/WebQA). Going to pull chat.py and take Wonderer for a spin today!
Thanks for sharing access!
Best,
Andrew