Honest evaluation methodology, the 20% vocabulary tax, and unanswerability mechanics in 31M tool-augmented SLMs
Hi Textile Labs team,
Documenting live failure modes so transparently (the ~30% held-out reading reality, Row I at 0/5, and Row E at 2/5) while training a 31.5M model from scratch on dual Kaggle T4s is exceptional open science. Seeing reading circuits emerge specifically after raising Muon LR to 0.025 on the 2D matrices is a fantastic empirical finding.
Looking at your architectural accounting (16 layers, 384 hidden dim, 16k vocab) and the benchmark dynamics:
The Softmax conservation dilemma in unanswerability (Row I: 0/5):
The reason the model cannot admit when a search result lacks the answer comes down to standard Softmax normalization. Because attention weights strictly sum to 1.0 across the sequence, the network is mathematically forced to allocate its probability mass over whatever tokens are present in the search snippet.
Without an explicit rejection sink or gating mechanism, background tokens absorb residual attention mass, compelling the generation head to extract an answer even from irrelevant text.The 20.0% vocabulary tax on a 31.5M budget:
With a 16,384 vocabulary at hidden size 384 and tied weights, your embedding table consumes ~6.29M parameters.
Out of your 31.47M budget, that single static lookup table accounts for exactly 20.0% of the entire model.
At hidden dimension 384 with GQA (6 heads / 2 KV), a complete transformer block costs roughly ~1.57M parameters. Your static embedding weights equal the parameter budget of 4 full transformer layers.
Applying two-stage low-rank factorization (16,384 -> 64 -> 384 = ~1.07M params) reclaims over 5.2M parameters. Reallocating those weights directly into depth would allow you to expand from 16 to 19 or 20 physical layers within the exact same 31.5M ceiling, providing substantial relational capacity for parsing noisy prose.Context retention and state continuity in tool loops (Row E: 2/5):
In multi-turn tool harnesses, passing raw previous turns and search outputs through dense quadratic attention quickly saturates the 1,024 context window and causes attention dilution.
In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore sub-100M efficiency using a 3:1 macro-topology (DGDA linear recurrence paired with sparse attention):
Instead of re-attending across historical search passes, linear recurrence updates an associative memory matrix in constant O(1) state space. This preserves conversation continuity and tool states across multi-turn queries without context bloat or quadratic slowdown.The 10-epoch curriculum and held-out divergence:
Your tuning vs held-out gap (11/20 down to 6/20) aligns closely with small-model memorization dynamics. In sub-50M regimes, repeating data across 10 epochs tends to drive cross-entropy down while causing zero-shot generalization on unseen syntax to plateau or drift. Fresh token diversity usually yields a much sharper reading threshold than multi-epoch recycling.
Did you observe gradient norm spikes or representation drift during the 10-epoch run when Muon was active on the attention projections?
Best,
Andrew
Thanks Andrew. This is genuinely useful and you clearly read the card closely.
You are completely right about the vocabulary point, and the math checks out. The 16,384 token embedding takes up about 6.3 million parameters, which is a flat 20% of the budget or roughly four layers' worth of capacity. Factorizing it or simply shrinking the vocabulary to buy depth is a real lever. We will test it in the next generation since it lines up perfectly with the work we are already doing to add depth.
I also agree that Row I is a genuine weakness when it comes to the model admitting a result does not contain the answer. The main cause for us right now is the data. We shipped with very few examples of unanswerable prompts, so the model never learned how to make that move. We have added those types of rows to the training data for the next model and expect Row I to improve. Your idea about attention sinks and explicit rejection is really interesting too, and we will keep it in mind if the data fix is not enough.
I really appreciate the quantitative feedback. This is exactly the kind of thing that makes releasing openly worth it.
If you have other ideas, whether on architecture, training data, or how we evaluate these models, we'd genuinely welcome them. Feedback like yours directly shapes what we build next, and we would love to hear anything you would like to share.
Thanks,
The textilelabs team
Hi Textile Labs team,
Glad the vocab math clicked! Gaining 4 extra layers on a 31M budget is going to be huge for depth.
A couple quick thoughts since you asked:
- For unanswerability: adding negative pairs usually does the trick. If it still hallucinates, try adding a dedicated null/sink token at prompt start. It gives Softmax a zero-penalty place to dump attention mass when the text has no answer.
- For tied factorization: when projecting down (e.g. 16k -> 64 -> 384), project back to rank-64 right before the LM head. That way you keep weights tied with the embedding table cleanly.
Really looking forward to Loom Weave 4! :)
Best,
Andrew