Hindi
Marathi
English
tokenizers
tokenizer
bpe
hinglish
minglish
code-mixed
indic
nlp
research-paper
akshar-32k / results

Commit History

Initial release of Akshar: The High-Efficiency Desi Tokenizer
0c87312
verified

Sujalvc commited on