Upload folder using huggingface_hub

README.md

@@ -0,0 +1,24 @@
+---
+language: [ro, en]
+license: odc-by
+tags: [romanian, tokenizer, bpe, diacritics]
+---
+
+# rost-tok-ro-normalized
+
+Romanian BPE tokenizer, 32,768 vocab, trained on cedilla-normalized text.
+
+## Measured fertility
+
+| tokenizer | tokens/word on Romanian |
+|---|---|
+| English-only (control) | 2.8421 |
+| Romanian, raw (mixed encodings) | 1.5660 |
+| **Romanian, normalized** | **1.5571** |
+| Romanian, diacritics stripped | 1.5190 — but **2.0083** on correct Romanian |
+
+A Romanian tokenizer is worth **1.8×** over an English one. Stripping
+diacritics looks 2.5% better on de-diacriticised text and is **29% worse** on
+correct Romanian, because it byte-falls-back on every `ș ț ă â î`.
+
+See https://github.com/StefanIancu/rost for the full experiment.

token_bytes.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7dd92e18c39f75b7499c2a15c5abc35e1a8620551b080000d43225d3366b9713
+size 132677

tokenizer.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:570dc39d4b1c202547c7632cb799340159176e026757c657921d805c5a06ce84
+size 410046