README.md

---
language:
- az
license: apache-2.0
library_name: sentencepiece
tags:
- azerbaijani
- tokenizer
- sentencepiece
- unigram
- low-resource
- turkic
---

# az_unigram_32k — Azerbaijani SentencePiece tokenizer (32K, Unigram)

A purpose-built **32,000-vocab SentencePiece Unigram** tokenizer for **Latin-script Azerbaijani**.

**On par with the best purpose-built Azerbaijani tokenizers at equal or smaller vocab — and ~2× more
token-efficient than the Turkish tokenizers people actually reuse for Azerbaijani.** Against
general-purpose tokenizers the gap is larger still: GPT-4 needs **~141% more tokens** for the same
Azerbaijani text (2.4×) — higher cost and effectively less usable context.

![Azerbaijani tokenizer comparison](az_tokenizer_comparison.png)

## Compared to other Azerbaijani & Turkic tokenizers (the real benchmark)

Beating GPT-4 on Azerbaijani is table stakes — *any* Azerbaijani-specific tokenizer does. The honest
question is how we stack up against tokenizers actually **built for**, or **reused on**, Azerbaijani.
Measured on **FLORES-200 Azerbaijani dev** (997 human-translated sentences — neutral for every tokenizer,
none trained on it; lower fertility is better):

| tokenizer | category | vocab | fertility (tok/word) | bytes/token |
|---|---|---:|---:|---:|
| LocalDoc az-en (Unigram) | Azerbaijani | 50,000 | 1.450 | 5.97 |
| aLLMA-2 (allmalab) | Azerbaijani | 32,000 | 1.522 | 5.68 |
| **az_unigram_32k (ours)** | **Azerbaijani** | **32,000** | **1.544** | **5.60** |
| XLM-R | Multilingual | 250,002 | 1.815 | 4.76 |
| NLLB-200 | Multilingual | 256,204 | 2.104 | 4.11 |
| mT5 | Multilingual | 250,100 | 2.362 | 3.66 |
| GPT-4o (o200k_base) | General-purpose | 200,019 | 2.383 | 3.63 |
| Turkish GPT-2 (ytu-cosmos) | Turkic / Az-tuned | 50,257 | 3.214 | 2.69 |
| BERTurk (Turkish, cased) | Turkic / Az-tuned | 32,000 | 3.223 | 2.68 |
| Llama 3 | General-purpose | 128,000 | 3.406 | 2.54 |
| mGPT-az (ai-forever) | Turkic / Az-tuned | 100,000 | 3.522 | 2.46 |
| Qwen2.5 | General-purpose | 151,643 | 3.562 | 2.43 |
| GPT-4 (cl100k_base) | General-purpose | 100,277 | 3.716 | 2.33 |

**Reading it honestly:**

- **Statistical tie with aLLMA-2 at equal 32K vocab** (1.544 vs 1.522 — a 1.4% gap). aLLMA-2 is the
  tokenizer from the *"Open foundation models for Azerbaijani"* line of work (WordPiece/BPE, 32K).
- **LocalDoc's small edge is mostly its 56%-larger vocab** (50K vs 32K). Fertility drops with vocab; a
  controlled sweep on this tokenizer showed 32K→48K ≈ −3.9%, so a 48–50K build of ours lands on top of it.
- **~2× ahead of both Turkish tokenizers** (BERTurk, Turkish GPT-2) and **2.3× ahead of Azerbaijani-adapted
  mGPT**. Takeaway: you **cannot** reuse a Turkish tokenizer for Azerbaijani, and shipping an "Azerbaijani
  model" on a stock multilingual tokenizer (like mGPT-az) leaves ~2× efficiency on the table.
- **A deliberate tradeoff explains the rest of the gap.** We set `split_digits=True` — every digit is its
  own token, for cleaner numeric/arithmetic behavior — which aLLMA-2 and LocalDoc do **not**. On text with
  numbers this costs us fertility *on purpose* (e.g. `2025` → 4 tokens, not 1). Back it out and ours **leads
  the Azerbaijani field** (~1.67 vs aLLMA-2 1.77 / LocalDoc 1.73 on in-domain web text). Our subword modeling
  is competitive-to-better; the fertility we spend is bought numeric behavior. All three preserve casing and
  the dotted/dotless-i distinction — no one wins by lowercasing.

## vs. general-purpose tokenizers (the cost hook)

Same FLORES eval, expressed as "how many more tokens the general tokenizers need vs ours":

| tokenizer | vocab | fertility (tok/word) | vs ours |
|---|---:|---:|---:|
| **az_unigram_32k (ours)** | **32,000** | **1.544** | — |
| XLM-R | 250,002 | 1.815 | 1.18× |
| GPT-4o (o200k_base) | 200,019 | 2.383 | 1.54× |
| Llama 3 | 128,000 | 3.406 | 2.21× |
| GPT-4 (cl100k_base) | 100,277 | 3.716 | **2.41×** |

We beat the massively-multilingual SentencePiece tokenizers (XLM-R, NLLB, mT5) **with ~8× smaller vocab**,
and the margin is *larger* on clean text than on web text — purpose-built morpheme segmentation shows most
where general tokenizers fragment Azerbaijani words. Reproduce everything with
`tokenizer/compare_fertility.py` (add `--include-az` for the Azerbaijani/Turkic set — on by default).

## Design

- **Model:** SentencePiece Unigram, vocab 32,000, `character_coverage=1.0`, NFKC, `byte_fallback=True`
  (zero `<unk>`), `split_digits=True`.
- **Script:** Latin-only (Modern North Azerbaijani). Casing preserved — including the meaningful
  dotted/dotless-i distinction (`i`/`İ` vs `ı`/`I`), which is **not** collapsed.
- **Byte-fallback rate:** 0.18% of tokens on held-out text (very low — clean coverage).
- **Specials:** `<unk>`=0, `<s>`=1, `</s>`=2, `<pad>`=3, plus `<|endoftext|>` (doc separator).
- **Training data:** ~415 MB sample of a cleaned, deduplicated Azerbaijani corpus (CulturaX + HPLT v2 +
  mC4 + Wikipedia). See the companion dataset card.

## Usage

```python
import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="az_unigram_32k.model")
ids = sp.encode("Azərbaycan dili türk dilləri ailəsinə aiddir.", out_type=int)
print(len(ids), sp.decode(ids))
```

## Limitations

- **Latin Azerbaijani only** — Cyrillic (legacy) and Perso-Arabic (South Azerbaijani) are out of scope.
- Trained on web-heavy text; rare technical/scientific vocabulary may segment less efficiently.
- `split_digits=True` trades raw fertility on number-heavy text for cleaner numeric behavior (a deliberate
  choice — see the comparison above).
- Fertility numbers are vs the listed tokenizer versions at eval time.

## FAQ

**How does this compare to other Azerbaijani tokenizers (not just GPT-4)?** It's a statistical tie with
aLLMA-2 at the same 32K vocab, within reach of LocalDoc's 50K (the difference is mostly its larger vocab),
and ~2× more efficient than Turkish tokenizers (BERTurk, Turkish GPT-2) or an Azerbaijani-tuned model that
kept a multilingual tokenizer (mGPT-az). See the table above.

**Why is it better than GPT-4's tokenizer?** It's purpose-built for Azerbaijani, so it encodes the same
text in ~2.4× fewer tokens → cheaper, more text per context window, faster. General tokenizers (GPT-4,
Llama, Qwen) are trained mostly on English/Chinese; Azerbaijani is a rounding error in their data, so they
**shatter words into tiny fragments** — sometimes raw bytes for ə/ğ/ı/ş/ç/ö/ü. Two mechanisms: (1)
Azerbaijani is agglutinative (*ev → evlər → evlərimizdə*) and we learned those stems+suffixes as units;
(2) our 32K vocab is spent *entirely* on Azerbaijani vs GPT-4's 100K split across 100+ languages.

**Is the benchmark fair?** Yes — measured on **FLORES-200** (human-translated; none of the tokenizers
trained on it specifically). Neutral ground, fully reproducible (`compare_fertility.py`). Every tokenizer
is run with `add_special_tokens=False`, and all Azerbaijani comparators preserve casing.

**Honest caveat:** better *for Azerbaijani specifically* — not for multilingual text or code, Latin-script
only.

## Citation

Part of an open Azerbaijani LLM stack (tokenizer → dataset → model → evals). Formal citation to follow.