| tokenizer | vocab | fertility (tok/word) | bytes/token |
|---|---|---|---|
| Ours (az_unigram_32k) | 32,000 | 1.881 | 4.575 |
| XLM-R | 250,002 | 1.958 | 4.395 |
| NLLB-200 | 256,204 | 2.329 | 3.696 |
| mT5 | 250,100 | 2.412 | 3.569 |
| GPT-4o (o200k_base) | 200,019 | 2.565 | 3.356 |
| Llama 3 (mirror) | 128,000 | 3.334 | 2.582 |
| Qwen2.5 | 151,643 | 3.555 | 2.421 |
| GPT-4 (cl100k_base) | 100,277 | 3.622 | 2.377 |