Numerical validation rigor, the 17.8% vocabulary tax, and discourse-level perplexity cliffs in 46M SLMs
Hi superAVTR,
Documenting the native PyTorch to Hugging Face Llama conversion with exact logit validation (cosine similarity 1.0000000000 and max absolute difference 9.54e-6) is exceptional systems engineering. Very few authors verify token-for-token numerical parity so thoroughly when bridging custom architectures into transformers.
Looking at your architectural schedule (12 layers, 512 hidden size, 16k vocab) and the benchmark results:
The 17.8% vocabulary tax on deep-thin allocation:
With a 16,000 SentencePiece vocabulary at hidden dimension 512, your tied embedding matrix consumes 8,192,000 parameters.
Out of your 45.9M parameter budget, that single static lookup table accounts for exactly 17.8% of the entire network.
At hidden dimension 512 with SwiGLU (intermediate 1,365) and 8 attention heads, a single transformer layer costs roughly 3.15M parameters. Your static embedding weights equal the parameter budget of more than 2.6 full transformer blocks.
Applying a two-stage low-rank factorization (16,000 -> 64 -> 512 = ~1.06M params) would reclaim over 7.1M parameters. Reallocating that reclaimed weight into active depth would allow expanding from 12 to 14 physical layers within the exact same 46M envelope, giving the network higher relational capacity.The LAMBADA perplexity cliff (2,539.74 PPL / 5.38% acc):
Your ARC-Easy score (36.11% acc_norm) confirms genuine factual signal above random baseline. However, the steep drop on LAMBADA highlights the boundary of a 12-layer stack on discourse-level continuation.
Because LAMBADA requires tracking narrative state across multi-sentence paragraphs to predict a target final word, 12 physical layers at hidden dimension 512 struggle to retain distant subject-verb constraints without deeper composition or recurrent state grounding.Discourse retention via recurrent fast-weights:
In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore sub-100M efficiency using a 3:1 macro-topology (DGDA linear recurrence paired with latent sparse attention):
Instead of relying strictly on shallow quadratic self-attention across 1,024 tokens, linear recurrence updates an associative memory matrix in constant O(1) state space.
This prevents trailing tokens from washing out early discourse context, improving multi-sentence continuation while keeping decode latency flat and slashing KV-cache overhead.
If you are planning a second iteration or testing deeper configurations for Tinizong, checking out the parameter allocation layout in the Maba v2 repo might provide some useful structural points for sub-50M topologies.
How many total tokens did the model see during pretraining, and were the Wikipedia articles concatenated with explicit document delimiters or chunked into isolated 512-token segments?
Best,
Andrew
Hi Andrew,
Thank you a lot for your reply!
Here some more details:
- this model is the 3rd model in a row of experimental models built with the purpose of understanding LLM's internals and learning how to build a full stack of LLM tech. Probably I have taken the hardest path for this, instead of just starting to finetune some existing open models
- this model (and the previous ones) were vibe coded with ChatGPT 5.* and all code cross-checked by me and Claude Sonnet and MoonShot Kimi. I did that cross-checking with Sonnet and Kimi since I am not an AI engineer. My background is in sys-admin/cloud/general development/systems security, not AI or deeplearning and I may bring bugs in the code.
- the logit validation and testing was suggested by GPT-6-Astra. The purpose was to ensure I am not exporting to HuggingFace some model that looks like working, but actually generates a corrupted output.
- the purpose of this model was to research if adding some synthetic dataset to the pretraining data, helps the model to improve output. And it does.
And here a reply for your questions:
- the model was trained on a dataset of 2.2B tokens + 20M synthetic tokens. The 2.2B is based on the dataset https://huggingface.co/datasets/lucadiliello/wikipedia_512_pretraining. The 20M synthetic dataset was generated by an LLM, based on a list of factCards created by me
- the documents were concatenated with a document delimiter (you can see that in the output, it is <|endoftext|>)
I will take later this week a look at Maba V2!
Thank again,
Adrian
Hi Adrian,
Coming from sysadmin and systems security and choosing the hard path to build full-stack from scratch rather than just finetuning someone else's model is the absolute best way to learn how this stuff actually works. That engineering background really shows in the logit validation check!
The 20M synthetic factCards explain a lot. That's why your ARC-Easy factual score held up so well at 36% on just 2.2B tokens. High-density synthetic facts punch way above their weight in sub-50M models.
Whenever you get around to checking out Maba v2, let me know what you think or feel free to ping me if you want to bounce ideas around for the next run :)
Best,
Andrew