shaikhsalman/zabaanai-pakistan-multilingual

🇵🇰 ZabaanAI - Pakistan Multilingual Language Model

An open-source, CPU-friendly AI model for Pakistan's major languages

🌐 Supported Languages

Language	Script	Coverage
Urdu	Arabic (Nastaliq)	✅ News, poetry, social media
Punjabi (Shahmukhi)	Arabic	✅ News, literature, transliteration
Sindhi	Arabic	✅ Large corpus (505M tokens)
Saraiki	Arabic	⚠️ Augmented from related languages
Balochi	Arabic	✅ Small curated datasets
Pashto	Arabic	✅ ZamAI corpus, textbooks, news
Hindko	Arabic	⚠️ Augmented from related languages
Brahui	Arabic	⚠️ Bridged from Balochi
Kashmiri	Arabic	✅ Dedicated corpora + news
English	Latin	✅ Wikipedia + parallel corpora
Roman Urdu	Latin	✅ Social media, code-mixed

🏗️ Base Model: google/mt5-small

We chose mT5-small over XLM-R and mBERT because it's the only architecture that natively supports all required tasks:

Requirement	mT5-small ✅	XLM-R ❌	mBERT ❌
Text Generation	✅ Native seq2seq	Encoder-only	Encoder-only
Translation	✅ Native	Can't generate	Can't generate
Summarization	✅ Native	Can't generate	Can't generate
Chatbot/QA	✅ Native	Can't generate	Can't generate
Sentiment	✅ Text-to-text	Native	Native
CPU Efficiency	✅ 47% lower latency*	Good	Good

*Encoder-decoder achieves 47% lower first-token latency and 4.7x higher throughput on CPU vs decoder-only (Microsoft 2025).

🎯 Supported Tasks

All tasks use mT5's unified text-to-text format:

Task	Input Format	Example Output
Text Generation	generate in Urdu: {prompt}	"پاکستان ایک خوبصورت ملک ہے..."
Translation	translate English to Urdu: {text}	"پاکستان ایک خوبصورت ملک ہے"
Summarization	summarize: {long_text}	"Short summary..."
Sentiment	sentiment: {text}	"positive" / "negative" / "neutral"
QA	question: {Q} context: {C}	"Answer text..."
Chatbot	conversation: user: {msg} assistant:	"Response text..."

📂 Repository Structure

├── scripts/
│   ├── 01_collect_datasets.py        # Download 35+ HF datasets
│   ├── 02_clean_text.py              # Clean, deduplicate, normalize
│   ├── 03_train_tokenizer.py         # Extend mT5 tokenizer
│   ├── 04_prepare_training_data.py  # T5 span corruption prep
│   ├── 05_train_cpu.py               # CPU-optimized LoRA pretraining
│   ├── 06_finetune_lora.py           # LoRA fine-tuning (5 tasks)
│   ├── 07_evaluate.py                # BLEU, ROUGE, perplexity
│   ├── 08_quantize.py                # ONNX INT8 / 8-bit quantization
│   └── 09_deploy_space.py           # HF Space deployment
├── docs/
│   ├── 01_DATASET_COLLECTION.md      # Full dataset sourcing guide
│   ├── 02_TOKENIZER_GUIDE.md         # Tokenizer improvement strategy
│   ├── 03_TRAINING_GUIDE.md          # CPU training configuration
│   ├── 04_FINETUNING_GUIDE.md        # LoRA fine-tuning recipes
│   └── 05_DEPLOYMENT_GUIDE.md        # 5 deployment options
├── deployment/
│   ├── app.py                        # Gradio web app (6 tasks)
│   ├── api_server.py                 # FastAPI REST API
│   ├── Dockerfile                    # Docker container
│   └── requirements.txt              # All dependencies
├── notebooks/
│   └── 01_explore_datasets.ipynb     # Dataset visualization
├── tokenizer/                        # Extended tokenizer (250,141 vocab)
├── config.yaml                       # Central configuration
├── setup.py                          # Environment setup
├── QUICKSTART.md                     # 10-step quick start
└── README.md                         # This file

⚡ Quick Start

1. Install Dependencies

pip install transformers datasets tokenizers accelerate peft sentencepiece
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install protobuf tiktoken gradio optimum[onnxruntime]

2. Collect Datasets

python scripts/01_collect_datasets.py

3. Clean & Merge

python scripts/02_clean_text.py

4. Extend Tokenizer

python scripts/03_train_tokenizer.py

5. Prepare Training Data

python scripts/04_prepare_training_data.py

6. Start CPU Training

python scripts/05_train_cpu.py --max_steps 50000 --batch_size 2 --lora_r 64

7. Fine-Tune for Tasks (GPU recommended)

python scripts/06_finetune_lora.py --task translation --batch_size 8
python scripts/06_finetune_lora.py --task sentiment --batch_size 8

8. Deploy

python scripts/09_deploy_space.py --username your-username --space_name zabaanai-demo

🔧 CPU Training Configuration

Setting	Value	Why
Base Model	google/mt5-small	300M params, 250k vocab, seq2seq
LoRA Rank	64	15% of params trainable (45M)
Batch Size	2	Low for CPU RAM
Gradient Accumulation	32	Effective batch = 64
Learning Rate	2e-4	Higher LR works with LoRA
Max Steps	50,000	Minimum for meaningful adaptation
Warmup	500 steps	1% of total
Scheduler	Cosine	Smooth decay
Checkpointing	Every 2,000 steps	Auto-resume supported
Precision	FP32	CPU-safe (no fp16/bf16)
Gradient Checkpointing	Enabled	40-50% memory savings

Training time: ~70 hours for 50K steps on 8-core CPU, or ~3 hours on T4 GPU.

🎯 Task Fine-Tuning

Translation

python scripts/06_finetune_lora.py --task translation
# Datasets: Humair332/Vast-Urdu (10M pairs), Kashmiri-English, Pashto-English

Sentiment Analysis

python scripts/06_finetune_lora.py --task sentiment
# Datasets: community-datasets/roman_urdu, urdu_sentiment_corpus

Summarization

python scripts/06_finetune_lora.py --task summarization
# Datasets: News articles with headlines as pseudo-summaries

Question Answering

python scripts/06_finetune_lora.py --task qa
# Datasets: Sindhi encyclopedia, generated Q&A pairs

Chatbot / Conversational AI

python scripts/06_finetune_lora.py --task chatbot
# Datasets: Urdu-Instruct, Sindhi SFT, Pashto Alpaca

🚀 Deployment Options

Option	Best For	Setup	Cost
Hugging Face Spaces	Public demo	2 minutes	Free / Paid GPU
Local CPU Server	Privacy, on-premise	10 minutes	Free
Docker Container	Scalable cloud	15 minutes	Varies
FastAPI API	Production microservices	10 minutes	Varies
ONNX Quantized	Edge/embedded	5 minutes	Free

📊 Data Mixture for Pretraining

Urdu:           35%  (news, poetry, social media, Wikipedia)
Sindhi:         20%  (large corpus available)
Pashto:         15%  (ZamAI corpora)
Punjabi:        10%  (news, transliteration)
Roman Urdu:      8%  (social media, tweets)
English:         7%  (prevent catastrophic forgetting)
Kashmiri:        3%  (smaller corpus)
Balochi:       1.5%  (very limited data)
Saraiki/Hindko/Brahui: 0.5%  (augmented from related)

🔤 Tokenizer

mT5's native tokenizer already covers all Pakistan scripts with 250,000 tokens + byte fallback. We extended it with 52 Pakistan-specific tokens:

• Cities: لاہور, کراچی, اسلام آباد, پشاور, کوئٹہ
• Names: محمد, علی, فاطمہ
• Institutions: سپریم کورٹ, نیشنل اسمبلی
• Roman Urdu: bhai, yaar, theek

Vocabulary size: 250,141 (was 250,100)

📈 Evaluation

python scripts/07_evaluate.py --model_path training/checkpoints/final --task translation
# Metrics: BLEU (translation), ROUGE (summarization), Perplexity (generation)

💾 Quantization for Production

# ONNX INT8 (recommended for CPU)
python scripts/08_quantize.py --model_path training/checkpoints/final --method onnx

# 8-bit (for GPU)
python scripts/08_quantize.py --model_path training/checkpoints/final --method 8bit

Benefits: 60-70% smaller model, 2-3x faster inference on CPU.

🤝 Contributing

We welcome contributions! Priority areas:

• More datasets for Saraiki, Hindko, Brahui
• Better evaluation benchmarks
• Fine-tuned task-specific adapters
• Documentation translations

📜 License

Apache 2.0 (same as mT5 base model)

🙏 Acknowledgments

• Google Research for mT5
• ZamAI for Pashto datasets
• aakashMeghwar01 for Sindhi corpus
• Humair332 for Vast-Urdu parallel corpus
• Omarrran for Kashmiri datasets
• community-datasets for Roman Urdu sentiment data

📚 Citation

@software{zabaanai2025,
  title = {ZabaanAI: Pakistan Multilingual Language Model},
  author = {Open Source Contributors},
  year = {2025},
  url = {https://huggingface.co/shaikhsalman/zabaanai-pakistan-multilingual}
}

---

Built with ❤️ for Pakistan's languages 🇵🇰