Upload README.md with huggingface_hub

README.md

@@ -15,29 +15,69 @@ tags:
 # az_unigram_32k — Azerbaijani SentencePiece tokenizer (32K, Unigram)
 
 A purpose-built **32,000-vocab SentencePiece Unigram** tokenizer for **Latin-script Azerbaijani**.
-On neutral, human-translated text it is **~2.4× more token-efficient than GPT-4** for Azerbaijani — i.e.
-GPT-4 needs ~141% more tokens for the same text, which means higher cost and effectively less context.
 
-## Headline: fertility (tokens/word — lower is better)
-
-Measured on **FLORES-200 Azerbaijani dev** (997 human-translated sentences, *not* in training — neutral
-for every tokenizer compared):
-
-| tokenizer | vocab | fertility (tok/word) | bytes/token | vs ours |
-|---|---:|---:|---:|---:|
-| **az_unigram_32k (ours)** | **32,000** | **1.544** | **5.603** | — |
-| XLM-R | 250,002 | 1.815 | 4.764 | 1.18× |
-| NLLB-200 | 256,204 | 2.104 | 4.111 | 1.36× |
-| mT5 | 250,100 | 2.362 | 3.660 | 1.53× |
-| GPT-4o (o200k_base) | 200,019 | 2.383 | 3.629 | 1.54× |
-| Llama 3 | 128,000 | 3.406 | 2.539 | 2.21× |
-| Qwen2.5 | 151,643 | 3.562 | 2.428 | 2.31× |
-| GPT-4 (cl100k_base) | 100,277 | 3.716 | 2.327 | **2.41×** |
+**On par with the best purpose-built Azerbaijani tokenizers at equal or smaller vocab — and ~2× more
+token-efficient than the Turkish tokenizers people actually reuse for Azerbaijani.** Against
+general-purpose tokenizers the gap is larger still: GPT-4 needs **~141% more tokens** for the same
+Azerbaijani text (2.4×) — higher cost and effectively less usable context.
+
+![Azerbaijani tokenizer comparison](az_tokenizer_comparison.png)
+
+## Compared to other Azerbaijani & Turkic tokenizers (the real benchmark)
+
+Beating GPT-4 on Azerbaijani is table stakes — *any* Azerbaijani-specific tokenizer does. The honest
+question is how we stack up against tokenizers actually **built for**, or **reused on**, Azerbaijani.
+Measured on **FLORES-200 Azerbaijani dev** (997 human-translated sentences — neutral for every tokenizer,
+none trained on it; lower fertility is better):
+
+| tokenizer | category | vocab | fertility (tok/word) | bytes/token |
+|---|---|---:|---:|---:|
+| LocalDoc az-en (Unigram) | Azerbaijani | 50,000 | 1.450 | 5.97 |
+| aLLMA-2 (allmalab) | Azerbaijani | 32,000 | 1.522 | 5.68 |
+| **az_unigram_32k (ours)** | **Azerbaijani** | **32,000** | **1.544** | **5.60** |
+| XLM-R | Multilingual | 250,002 | 1.815 | 4.76 |
+| NLLB-200 | Multilingual | 256,204 | 2.104 | 4.11 |
+| mT5 | Multilingual | 250,100 | 2.362 | 3.66 |
+| GPT-4o (o200k_base) | General-purpose | 200,019 | 2.383 | 3.63 |
+| Turkish GPT-2 (ytu-cosmos) | Turkic / Az-tuned | 50,257 | 3.214 | 2.69 |
+| BERTurk (Turkish, cased) | Turkic / Az-tuned | 32,000 | 3.223 | 2.68 |
+| Llama 3 | General-purpose | 128,000 | 3.406 | 2.54 |
+| mGPT-az (ai-forever) | Turkic / Az-tuned | 100,000 | 3.522 | 2.46 |
+| Qwen2.5 | General-purpose | 151,643 | 3.562 | 2.43 |
+| GPT-4 (cl100k_base) | General-purpose | 100,277 | 3.716 | 2.33 |
+
+**Reading it honestly:**
+
+- **Statistical tie with aLLMA-2 at equal 32K vocab** (1.544 vs 1.522 — a 1.4% gap). aLLMA-2 is the
+  tokenizer from the *"Open foundation models for Azerbaijani"* line of work (WordPiece/BPE, 32K).
+- **LocalDoc's small edge is mostly its 56%-larger vocab** (50K vs 32K). Fertility drops with vocab; a
+  controlled sweep on this tokenizer showed 32K→48K ≈ −3.9%, so a 48–50K build of ours lands on top of it.
+- **~2× ahead of both Turkish tokenizers** (BERTurk, Turkish GPT-2) and **2.3× ahead of Azerbaijani-adapted
+  mGPT**. Takeaway: you **cannot** reuse a Turkish tokenizer for Azerbaijani, and shipping an "Azerbaijani
+  model" on a stock multilingual tokenizer (like mGPT-az) leaves ~2× efficiency on the table.
+- **A deliberate tradeoff explains the rest of the gap.** We set `split_digits=True` — every digit is its
+  own token, for cleaner numeric/arithmetic behavior — which aLLMA-2 and LocalDoc do **not**. On text with
+  numbers this costs us fertility *on purpose* (e.g. `2025` → 4 tokens, not 1). Back it out and ours **leads
+  the Azerbaijani field** (~1.67 vs aLLMA-2 1.77 / LocalDoc 1.73 on in-domain web text). Our subword modeling
+  is competitive-to-better; the fertility we spend is bought numeric behavior. All three preserve casing and
+  the dotted/dotless-i distinction — no one wins by lowercasing.
+
+## vs. general-purpose tokenizers (the cost hook)
+
+Same FLORES eval, expressed as "how many more tokens the general tokenizers need vs ours":
+
+| tokenizer | vocab | fertility (tok/word) | vs ours |
+|---|---:|---:|---:|
+| **az_unigram_32k (ours)** | **32,000** | **1.544** | — |
+| XLM-R | 250,002 | 1.815 | 1.18× |
+| GPT-4o (o200k_base) | 200,019 | 2.383 | 1.54× |
+| Llama 3 | 128,000 | 3.406 | 2.21× |
+| GPT-4 (cl100k_base) | 100,277 | 3.716 | **2.41×** |
 
 We beat the massively-multilingual SentencePiece tokenizers (XLM-R, NLLB, mT5) **with ~8× smaller vocab**,
 and the margin is *larger* on clean text than on web text — purpose-built morpheme segmentation shows most
-where general tokenizers fragment Azerbaijani words. (On in-domain web text: ours 1.881 vs GPT-4 3.622.)
-Reproduce with `tokenizer/compare_fertility.py`.
+where general tokenizers fragment Azerbaijani words. Reproduce everything with
+`tokenizer/compare_fertility.py` (add `--include-az` for the Azerbaijani/Turkic set — on by default).
 
 ## Design
 
@@ -63,27 +103,30 @@ print(len(ids), sp.decode(ids))
 
 - **Latin Azerbaijani only** — Cyrillic (legacy) and Perso-Arabic (South Azerbaijani) are out of scope.
 - Trained on web-heavy text; rare technical/scientific vocabulary may segment less efficiently.
+- `split_digits=True` trades raw fertility on number-heavy text for cleaner numeric behavior (a deliberate
+  choice — see the comparison above).
 - Fertility numbers are vs the listed tokenizer versions at eval time.
 
-## FAQ — why is this better than GPT-4's tokenizer?
+## FAQ
 
-**Short version:** it's purpose-built for Azerbaijani, so it encodes the same text in ~2.4× fewer tokens
-than GPT-4 → cheaper, more text per context window, faster.
+**How does this compare to other Azerbaijani tokenizers (not just GPT-4)?** It's a statistical tie with
+aLLMA-2 at the same 32K vocab, within reach of LocalDoc's 50K (the difference is mostly its larger vocab),
+and ~2× more efficient than Turkish tokenizers (BERTurk, Turkish GPT-2) or an Azerbaijani-tuned model that
+kept a multilingual tokenizer (mGPT-az). See the table above.
 
-**Why (the mechanism):** general tokenizers (GPT-4, Llama, Qwen) are trained mostly on English/Chinese;
-Azerbaijani is a rounding error in their data, so they never learned Azerbaijani subwords and **shatter
-words into tiny fragments** — sometimes raw bytes for ə/ğ/ı/ş/ç/ö/ü. Two concrete reasons we win:
-1. **Agglutinative morphology** — Azerbaijani stacks suffixes (*ev → evlər → evlərimizdə*); we learned those
-   stems+suffixes as units, general tokenizers fragment every word.
-2. **Vocabulary budget** — our 32K vocab is spent *entirely* on Azerbaijani, vs GPT-4's 100K / XLM-R's 250K
-   split across 100+ languages. Focused-and-small beats huge-and-diluted: we beat the 250K multilingual
-   tokenizers with **8× less vocab**.
+**Why is it better than GPT-4's tokenizer?** It's purpose-built for Azerbaijani, so it encodes the same
+text in ~2.4× fewer tokens → cheaper, more text per context window, faster. General tokenizers (GPT-4,
+Llama, Qwen) are trained mostly on English/Chinese; Azerbaijani is a rounding error in their data, so they
+**shatter words into tiny fragments** — sometimes raw bytes for ə/ğ/ı/ş/ç/ö/ü. Two mechanisms: (1)
+Azerbaijani is agglutinative (*ev → evlər → evlərimizdə*) and we learned those stems+suffixes as units;
+(2) our 32K vocab is spent *entirely* on Azerbaijani vs GPT-4's 100K split across 100+ languages.
 
-**Is the benchmark fair?** Yes — measured on **FLORES-200** (human-translated; *none* of the tokenizers
-trained on it specifically). Neutral ground, fully reproducible (`compare_fertility.py`).
+**Is the benchmark fair?** Yes — measured on **FLORES-200** (human-translated; none of the tokenizers
+trained on it specifically). Neutral ground, fully reproducible (`compare_fertility.py`). Every tokenizer
+is run with `add_special_tokens=False`, and all Azerbaijani comparators preserve casing.
 
-**Honest caveat:** better *for Azerbaijani specifically* — not for multilingual text or code, and
-Latin-script only.
+**Honest caveat:** better *for Azerbaijani specifically* — not for multilingual text or code, Latin-script
+only.
 
 ## Citation