Fill-Mask
Transformers
Safetensors
Chinese
bert
chinesebabylm
masked-language-modeling
baby-lm
control
adamw
# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("LikC1606/chinesebabylm-control-adamw-bert-mlm-9p53m")
model = AutoModelForMaskedLM.from_pretrained("LikC1606/chinesebabylm-control-adamw-bert-mlm-9p53m", device_map="auto")Quick Links
ChineseBabyLM Control AdamW BERT MLM 9.53M
Small from-scratch BERT-style masked language model trained as the Round 20 DeepScientist control-arm AdamW no-bias-correction pilot.
Model Details
- Architecture:
BertForMaskedLM - Model type: masked language model / encoder-only BERT
- Parameters: 9,527,714
- Layers: 4
- Attention heads: 6
- Hidden size: 384
- Vocabulary size: 5,666
- Max position embeddings: 256
Training Data
Chinese BabyLM official 10k stratified sample; control-arm Round 20 AdamW no-bias-correction pilot.
Evaluation
Official zero-shot evaluator results:
| Task | Accuracy |
|---|---|
| ZhoBLiMP | 59.74 |
| Hanzi structure | 47.00 |
| Hanzi pinyin | 31.00 |
| Mean available tasks | 45.91 |
Package Source
This model was packaged from control_round20_adamw_no_bias_correction_bert_mlm_9p53m in the local ChineseBabyLM best-model archive.
- Downloads last month
- 5
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="LikC1606/chinesebabylm-control-adamw-bert-mlm-9p53m")