---
pipeline_tag: fill-mask
language: deu
license: mit
tags:
  - trimmed
library_name: transformers
base_model: jhu-clsp/mmBERT-base
base_model_relation: quantized
datasets:
  - Lumberjackk/fineweb-2-trimming
---

# mmBERT-base-deu-32768
This model is a 55.86% smaller version of [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) optimized for German language via vocabulary size reduction using the [trimming](https://huggingface.co/blog/introduction-to-trimming) method.  
This trimmed model should perform similarly to the original model with only 32,768 tokens and a much smaller memory footprint. However, it may not perform well for other languages as tokens not commonly used in the selected languages were removed from the vocabulary.

## Model Statistics
| Metric | Original | Trimmed | Reduction |
|--------|----------|---------|-----------|
| **Vocabulary size** | 256,000 tokens | 32,768 tokens | **87.20%** |
| **Model size** | 306,939,648 params | 135,497,472 params | **55.86%** |

![image](https://cdn-uploads.huggingface.co/production/uploads/613b0a62a14099d5afed7830/3bAHdqRvu-haO_RxyOwVo.png)

## Mining Dataset Statistics
- **Number of texts used for mining**: 200,000 texts  
- **Dataset**: [Lumberjackk/fineweb-2-trimming](https://huggingface.co/datasets/Lumberjackk/fineweb-2-trimming)

## Usage
```python
from transformers import AutoModel, AutoTokenizer

model_name = "Lumberjackk/mmBERT-base-deu-32768"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
```

## Citation

#### mmBERT
```
@misc{marone2025mmbertmodernmultilingualencoder,
      title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning}, 
      author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
      year={2025},
      eprint={2509.06888},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.06888}, 
}
```