arisin/hindi-bpe-tokenizer

Hugging Face Deployment Guide

This guide explains how to publish the Hindi BPE tokenizer to the Hugging Face Hub and how end users can interact with it.

1. Prerequisites

• Install the Hugging Face CLI: pip install --upgrade huggingface_hub
• Authenticate: huggingface-cli login
• Ensure the tokenizer assets exist under hf_tokenizer/ (tokenizer.json + tokenizer_config.json). Generate them via:

python3 scripts/export_hf_tokenizer.py

2. Prepare Repository Structure

Create a directory you will push to the Hub, e.g.:

export HF_REPO=arisin/hindi-bpe-tokenizer
huggingface-cli repo create $HF_REPO --token YOUR_TOKEN --type space --private false

mkdir -p build/$HF_REPO
cp hf_tokenizer/tokenizer.json build/$HF_REPO/
cp hf_tokenizer/tokenizer_config.json build/$HF_REPO/
cp hf_tokenizer/special_tokens_map.json build/$HF_REPO/
cp docs/huggingface_guide.md build/$HF_REPO/README.md
cp notebooks/hf_tokenizer_demo.ipynb build/$HF_REPO/

3. Upload to Hugging Face Hub

cd build/$HF_REPO
git init
git add .
git commit -m \"Add Hindi BPE tokenizer\"
git remote add origin https://huggingface.co/$HF_REPO
git push -u origin main

Once pushed, the tokenizer is available at https://huggingface.co/$HF_REPO.

4. End-User Usage

Python Example

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(\"arnabsinha/hindi-bpe-tokenizer\")
encoded = tokenizer(\"वाराणसी की सुबह अद्भुत होती है।\")
print(encoded)
print(tokenizer.decode(encoded[\"input_ids\"]))

CLI Example

python -m transformers.models.auto.tokenization_auto \
  --tokenizer arnabsinha/hindi-bpe-tokenizer \
  --text \"मुंबई की लोकल ट्रेनें लाखों सपने ले जाती हैं।\"

Tokenizer Playground

On the model page, select the "Tokenizer" widget to interactively test tokenization in the browser.

5. Incorporating the Notebook

The notebook notebooks/hf_tokenizer_demo.ipynb can be uploaded alongside the tokenizer or converted to a README example to guide notebook-based exploration.

6. Updating the Tokenizer

• Rerun scripts/export_hf_tokenizer.py after retraining.
• Update the hub repository by repeating the upload steps.
• Optionally tag releases via git tag v1.1 before pushing for versioning.

This setup enables end users to consume the tokenizer via the Hugging Face ecosystem with minimal friction.