---
license: apache-2.0
language: en
tags: [babylm, babylm-2026, strict-small, deberta, vision-initialized]
---
# deberta-base-75k-sam — BabyLM 2026 strict-small

DeBERTa-v3-base trained from scratch (10 epochs, ≤100M words exposure),
75k byte-level BPE tokenizer. **Vision-initialized**: before training,
input embeddings of the 21,134 vocabulary tokens attested in visual
grounding data (Flickr30k Entities, RefCOCO/g/+, THINGS) were seeded from
SAM ViT-B region features (bbox-patch pooling, mean-centered,
L2-normalized, scaled); all other tokens use standard random init.
Training: lr 2e-4, batch 256, grad-acc 4, MLM 15%.

**Training data**: custom ~9.9M-word corpus (`bb24.train`) from our BabyLM
2024 submission ([Edman et al. 2024, "Are BabyLMs Second Language
Learners?"](https://aclanthology.org/2024.conll-babylm.14/)) — a mixture
of LLM-synthesized paraphrase/contrastive data (SynCSE-partial; Zhang et
al. 2021) and portions of the official BabyLM corpus (Simple Wikipedia,
Gutenberg, Switchboard). Within the strict-small 10M-word budget.

Intermediate checkpoints: revisions `chck_1M` … `chck_100M` (words seen)
and `step1000` … `step25740`; `main` = final.

Findings (vs the matched control
[`deberta-base-75k`](https://huggingface.co/augustinian-babylm/deberta-base-75k)):
consistent gains on object property knowledge (COMPS; and a purpose-built
visual-property benchmark, +1.9–3.0 pts across 3 seeds), no effect on
syntax. Code, benchmark, analyses:
https://github.com/bylinina/augustinian_babylm