| tokenizer | category | vocab | fertility (tok/word) | bytes/token |
|---|---|---:|---:|---:|
| LocalDoc az-en (unigram 50k) | Azerbaijani | 50,000 | 1.727 | 4.985 |
| aLLMA-2 (allmalab) | Azerbaijani | 32,000 | 1.771 | 4.861 |
| Ours (az_unigram_32k) | Azerbaijani | 32,000 | 1.881 | 4.575 |
| XLM-R | Multilingual | 250,002 | 1.958 | 4.395 |
| NLLB-200 | Multilingual | 256,204 | 2.329 | 3.696 |
| mT5 | Multilingual | 250,100 | 2.412 | 3.569 |
| GPT-4o (o200k_base) | General-purpose | 200,019 | 2.565 | 3.356 |
| BERTurk (Turkish 32k) | Turkic/Az-tuned | 32,000 | 3.117 | 2.761 |
| Turkish GPT-2 (ytu-cosmos) | Turkic/Az-tuned | 50,257 | 3.122 | 2.757 |
| Llama 3 (mirror) | General-purpose | 128,000 | 3.334 | 2.582 |
| mGPT-az (ai-forever) | Turkic/Az-tuned | 100,000 | 3.427 | 2.512 |
| Qwen2.5 | General-purpose | 151,643 | 3.555 | 2.421 |
| GPT-4 (cl100k_base) | General-purpose | 100,277 | 3.622 | 2.377 |