| tokenizer | category | vocab | fertility (tok/word) | bytes/token |
|---|---|---|---|---|
| LocalDoc az-en (unigram 50k) | Azerbaijani | 50,000 | 1.450 | 5.965 |
| aLLMA-2 (allmalab) | Azerbaijani | 32,000 | 1.522 | 5.684 |
| Ours (az_unigram_32k) | Azerbaijani | 32,000 | 1.544 | 5.603 |
| XLM-R | Multilingual | 250,002 | 1.815 | 4.764 |
| NLLB-200 | Multilingual | 256,204 | 2.104 | 4.111 |
| mT5 | Multilingual | 250,100 | 2.362 | 3.660 |
| GPT-4o (o200k_base) | General-purpose | 200,019 | 2.383 | 3.629 |
| Turkish GPT-2 (ytu-cosmos) | Turkic/Az-tuned | 50,257 | 3.214 | 2.690 |
| BERTurk (Turkish 32k) | Turkic/Az-tuned | 32,000 | 3.223 | 2.684 |
| Llama 3 (mirror) | General-purpose | 128,000 | 3.406 | 2.539 |
| mGPT-az (ai-forever) | Turkic/Az-tuned | 100,000 | 3.522 | 2.456 |
| Qwen2.5 | General-purpose | 151,643 | 3.562 | 2.428 |
| GPT-4 (cl100k_base) | General-purpose | 100,277 | 3.716 | 2.327 |