---
license: cc-by-4.0
language:
- en
library_name: transformers
pipeline_tag: fill-mask
tags:
- babylm
- deberta-v3
- masked-lm
- visual-grounding
- data-efficient-pretraining
- vision-init-sam
base_model:
- facebook/sam-vit-base
---

# deberta-base-100k-sam

**Vision-seeded** model. Before training, input-embedding rows for words that label regions in visual grounding data were set to the average of SAM ViT-B (Kirillov et al. 2023) features over those regions; all other rows keep their random initialization. Seeded vectors are mean-centered, L2-normalized, and scaled to the model's initializer standard deviation.

Baseline: `augustinian-babylm/deberta-base-100k`.

Part of the model set for *Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model* ([paper](https://openreview.net/forum?id=B4TD4XdlwF), [code](https://github.com/bylinina/augustinian_babylm)).

## Usage

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("augustinian-babylm/babylm-bpe-100k")
model = AutoModelForMaskedLM.from_pretrained("augustinian-babylm/deberta-base-100k-sam")
```

## Training

DeBERTa-v3-base masked LM (768 hidden, 12 layers, 12 heads), trained for 10 epochs
on `bb24.train`, the ~9.9M-word corpus of Edman et al. (2024), which mixes
LLM-generated paraphrase data with portions of the official BabyLM corpus. This is
**not** the official 2026 strict-small distribution.

| | |
|---|---|
| Learning rate | 2e-4, cosine schedule, 4000 warmup steps |
| Optimizer | AdamW (beta1 0.9, beta2 0.95), weight decay 0.01 |
| Effective batch | 256 (gradient accumulation 4) |
| Context length | 64 to 128 from epoch 5, max 512 |
| Vocabulary | BPE, 100k merges (`augustinian-babylm/babylm-bpe-100k`) |

Only the input-embedding initialization differs between a vision-init model and its
baseline. Every other hyperparameter, the data order, and the random seed are held
fixed.

## Checkpoints

Intermediate checkpoints are stored as branches, saved Pythia-style by step:

```python
from transformers import AutoModelForMaskedLM
AutoModelForMaskedLM.from_pretrained("augustinian-babylm/deberta-base-100k-sam", revision="step256")
```

Call `huggingface_hub.list_repo_refs("augustinian-babylm/deberta-base-100k-sam")` to list the available steps.

## Results

Vision seeding produces a consistent object-property gain (COMPS, +1.30 averaged
over nine encoder-by-vocabulary configurations, positive in all nine) while
grammar-focused benchmarks stay flat. On a corpus-tailored
Visual-Property Swap probe the seeded model leads the baseline in all three
training seeds; a pair-level re-analysis shows that the advantage cannot be
attributed to the individual seeded words. Details:
https://github.com/bylinina/augustinian_babylm

## Limitations

Trained on a non-standard corpus (see above), so results are not directly comparable
to submissions trained on the official 2026 strict-small data. Single masked-LM
architecture. Research artifact, not intended for deployment.

## Citation

```bibtex
@inproceedings{bylinina2026augustinian,
  title     = {Augustinian BabyLM: What Ostensive Definition Can and Cannot
               Teach a Small Language Model},
  author    = {Bylinina, Lisa},
  booktitle = {Proceedings of the BabyLM Workshop},
  year      = {2026},
  url       = {https://openreview.net/forum?id=B4TD4XdlwF}
}
```