Upload fertility_heldout.md with huggingface_hub

fertility_heldout.md

@@ -1,10 +1,15 @@
-| tokenizer | vocab | fertility (tok/word) | bytes/token |
-|---|---:|---:|---:|
-| Ours (az_unigram_32k) | 32,000 | 1.881 | 4.575 |
-| XLM-R | 250,002 | 1.958 | 4.395 |
-| NLLB-200 | 256,204 | 2.329 | 3.696 |
-| mT5 | 250,100 | 2.412 | 3.569 |
-| GPT-4o (o200k_base) | 200,019 | 2.565 | 3.356 |
-| Llama 3 (mirror) | 128,000 | 3.334 | 2.582 |
-| Qwen2.5 | 151,643 | 3.555 | 2.421 |
-| GPT-4 (cl100k_base) | 100,277 | 3.622 | 2.377 |
+| tokenizer | category | vocab | fertility (tok/word) | bytes/token |
+|---|---|---:|---:|---:|
+| LocalDoc az-en (unigram 50k) | Azerbaijani | 50,000 | 1.727 | 4.985 |
+| aLLMA-2 (allmalab) | Azerbaijani | 32,000 | 1.771 | 4.861 |
+| Ours (az_unigram_32k) | Azerbaijani | 32,000 | 1.881 | 4.575 |
+| XLM-R | Multilingual | 250,002 | 1.958 | 4.395 |
+| NLLB-200 | Multilingual | 256,204 | 2.329 | 3.696 |
+| mT5 | Multilingual | 250,100 | 2.412 | 3.569 |
+| GPT-4o (o200k_base) | General-purpose | 200,019 | 2.565 | 3.356 |
+| BERTurk (Turkish 32k) | Turkic/Az-tuned | 32,000 | 3.117 | 2.761 |
+| Turkish GPT-2 (ytu-cosmos) | Turkic/Az-tuned | 50,257 | 3.122 | 2.757 |
+| Llama 3 (mirror) | General-purpose | 128,000 | 3.334 | 2.582 |
+| mGPT-az (ai-forever) | Turkic/Az-tuned | 100,000 | 3.427 | 2.512 |
+| Qwen2.5 | General-purpose | 151,643 | 3.555 | 2.421 |
+| GPT-4 (cl100k_base) | General-purpose | 100,277 | 3.622 | 2.377 |