ChineseBabyLM Control AdamW BERT MLM 9.53M
Small from-scratch BERT-style masked language model trained as the Round 20 DeepScientist control-arm AdamW no-bias-correction pilot.
Model Details
- Architecture:
BertForMaskedLM - Model type: masked language model / encoder-only BERT
- Parameters: 9,527,714
- Layers: 4
- Attention heads: 6
- Hidden size: 384
- Vocabulary size: 5,666
- Max position embeddings: 256
Training Data
Chinese BabyLM official 10k stratified sample; control-arm Round 20 AdamW no-bias-correction pilot.
Evaluation
Official zero-shot evaluator results:
| Task | Accuracy |
|---|---|
| ZhoBLiMP | 59.74 |
| Hanzi structure | 47.00 |
| Hanzi pinyin | 31.00 |
| Mean available tasks | 45.91 |
Package Source
This model was packaged from control_round20_adamw_no_bias_correction_bert_mlm_9p53m in the local ChineseBabyLM best-model archive.
- Downloads last month
- 5