Upload fertility_flores.md with huggingface_hub

fertility_flores.md

@@ -1,10 +1,15 @@
-| tokenizer | vocab | fertility (tok/word) | bytes/token |
-|---|---:|---:|---:|
-| Ours (az_unigram_32k) | 32,000 | 1.544 | 5.603 |
-| XLM-R | 250,002 | 1.815 | 4.764 |
-| NLLB-200 | 256,204 | 2.104 | 4.111 |
-| mT5 | 250,100 | 2.362 | 3.660 |
-| GPT-4o (o200k_base) | 200,019 | 2.383 | 3.629 |
-| Llama 3 (mirror) | 128,000 | 3.406 | 2.539 |
-| Qwen2.5 | 151,643 | 3.562 | 2.428 |
-| GPT-4 (cl100k_base) | 100,277 | 3.716 | 2.327 |
+| tokenizer | category | vocab | fertility (tok/word) | bytes/token |
+|---|---|---:|---:|---:|
+| LocalDoc az-en (unigram 50k) | Azerbaijani | 50,000 | 1.450 | 5.965 |
+| aLLMA-2 (allmalab) | Azerbaijani | 32,000 | 1.522 | 5.684 |
+| Ours (az_unigram_32k) | Azerbaijani | 32,000 | 1.544 | 5.603 |
+| XLM-R | Multilingual | 250,002 | 1.815 | 4.764 |
+| NLLB-200 | Multilingual | 256,204 | 2.104 | 4.111 |
+| mT5 | Multilingual | 250,100 | 2.362 | 3.660 |
+| GPT-4o (o200k_base) | General-purpose | 200,019 | 2.383 | 3.629 |
+| Turkish GPT-2 (ytu-cosmos) | Turkic/Az-tuned | 50,257 | 3.214 | 2.690 |
+| BERTurk (Turkish 32k) | Turkic/Az-tuned | 32,000 | 3.223 | 2.684 |
+| Llama 3 (mirror) | General-purpose | 128,000 | 3.406 | 2.539 |
+| mGPT-az (ai-forever) | Turkic/Az-tuned | 100,000 | 3.522 | 2.456 |
+| Qwen2.5 | General-purpose | 151,643 | 3.562 | 2.428 |
+| GPT-4 (cl100k_base) | General-purpose | 100,277 | 3.716 | 2.327 |