license: apache-2.0
language: en
tags:
- babylm
- babylm-2026
- strict-small
- deberta
- vision-initialized
deberta-base-75k-sam — BabyLM 2026 strict-small
DeBERTa-v3-base trained from scratch (10 epochs, ≤100M words exposure), 75k byte-level BPE tokenizer. Vision-initialized: before training, input embeddings of the 21,134 vocabulary tokens attested in visual grounding data (Flickr30k Entities, RefCOCO/g/+, THINGS) were seeded from SAM ViT-B region features (bbox-patch pooling, mean-centered, L2-normalized, scaled); all other tokens use standard random init. Training: lr 2e-4, batch 256, grad-acc 4, MLM 15%.
Training data: custom ~9.9M-word corpus (bb24.train) from our BabyLM
2024 submission (Edman et al. 2024, "Are BabyLMs Second Language
Learners?") — a mixture
of LLM-synthesized paraphrase/contrastive data (SynCSE-partial; Zhang et
al. 2021) and portions of the official BabyLM corpus (Simple Wikipedia,
Gutenberg, Switchboard). Within the strict-small 10M-word budget.
Intermediate checkpoints: revisions chck_1M … chck_100M (words seen)
and step1000 … step25740; main = final.
Findings (vs the matched control
deberta-base-75k):
consistent gains on object property knowledge (COMPS; and a purpose-built
visual-property benchmark, +1.9–3.0 pts across 3 seeds), no effect on
syntax. Code, benchmark, analyses:
https://github.com/bylinina/augustinian_babylm