---
language: [ro, en]
license: odc-by
tags: [romanian, tokenizer, bpe, diacritics]
---

# rost-tok-ro-normalized

Romanian BPE tokenizer, 32,768 vocab, trained on cedilla-normalized text.

## Measured fertility

| tokenizer | tokens/word on Romanian |
|---|---|
| English-only (control) | 2.8421 |
| Romanian, raw (mixed encodings) | 1.5660 |
| **Romanian, normalized** | **1.5571** |
| Romanian, diacritics stripped | 1.5190 — but **2.0083** on correct Romanian |

A Romanian tokenizer is worth **1.8×** over an English one. Stripping
diacritics looks 2.5% better on de-diacriticised text and is **29% worse** on
correct Romanian, because it byte-falls-back on every `ș ț ă â î`.