ghananlpcommunity/ghana-english-ipa2text-tokenizer

Ghanaian-English IPA-to-Text Tokenizer

google-t5/t5-small's tokenizer, extended to also read Ghanaian-English IPA phoneme sequences — 80 new tokens added for phones the base vocabulary doesn't have (ɡ, tʃ, ŋ, etc.), taking it from 32,100 to 32,180 tokens. Ordinary English text still tokenizes identically to stock google-t5/t5-small (verified: byte-identical ids on English input).

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("ghananlpcommunity/ghana-english-ipa2text-tokenizer")

tok("The Governor raised the policy rate.").input_ids   # same ids as stock t5-small
tok("ɡ ʊ d m ɔː n ɪ ŋ").input_ids                        # phones tokenize cleanly, no <unk>

IPA input should be space-separated phones with punctuation kept as its own tokens (the format ghana-english-g2p and ghananlpcommunity/ghana-english-phoneme-asr both produce). Numbers should be spelled out as words before phonemising ("1,000" → "one thousand"), not phonemised digit-by-digit.

Used by

ghananlpcommunity/t5-small-ipa2text-v2 — fine-tuned with this tokenizer to convert Ghanaian-English IPA into text. Use that model's tokenizer copy (same repo either way) rather than mixing this with a different checkpoint.