Sanchari — Technical Blueprint (Full — Investor Grade)

Version: v0.1
Purpose: Complete technical blueprint to develop Sanchari-S → Sanchari-M → Sanchari-L
Target audience: engineers, infrastructure teams, investors

Summary (one line)

Build a practical, India-focused multilingual instruction-following LM family (S: ~200–350M, M: ~1–3B, L: 7B+) using modern efficient training (PyTorch + DeepSpeed/Accelerate + FlashAttention), with explicit data provenance, safety audits, and production deployment targets.

1. Design principles (what we deliver and why)

Practicality first — Sanchari-S must be cheap to train and fast to run for API/mobile use.
Indian language competence — prioritize English (Indian), Hindi, Telugu, mixed-script.
Safety & governance — every training stage includes PII scrubbing and red-team testing before any checkpoint release.
Modular pipeline — tokenizer, preprocessing, training config, adapter-based instruction tuning.
Deliverables at each phase — private checkpoints for investors (under NDA), evaluation reports, HF repo + demo.

2. Model architecture & approach

2.1 Base model family

Sanchari-S: ~200–350M parameters. Decoder-only transformer (GPT-like). Primary use: fast inference, API & mobile clients.
Sanchari-M: ~1–3B parameters. Better instruction following and multi-turn coherence.
Sanchari-L: ~7B+ parameters. Full foundation for enterprise applications.

2.2 Model topology (recommended)

Type: Decoder-only transformer.
Layer scaling (example recipes):
- Sanchari-S: 24 layers × 8 heads × 2048 hidden (example ≈ 300M)
- Sanchari-M: 36 layers × 16 heads × 4096 hidden (example ≈ 1.3B)
- Sanchari-L: 48–80 layers × 32 heads × 6144 hidden (approx 7B)
Attention: FlashAttention2 compatible (for GPU memory and speed).
Norms and embeddings: RMSNorm or LayerNorm, rotary positional embeddings (RoPE) for stable long-range.
Efficiency: Provide LoRA adapters and QLoRA options for cost-effective instruction tuning.

Rationale: decoder-only is industry standard for instruction-following and easier to deploy as an API.

3. Tokenizer & text processing

3.1 Tokenizer strategy

Tokenizer: SentencePiece Unigram or BPE with ~50k vocabulary, trained on mixed Indic + English corpus.
Features required:
- Mixed script normalization (normalize Unicode, NFKC)
- Preserve whitespace tokens for code-like text
- Subword segmentation for Indic scripts
Tooling: sentencepiece or tokenizers (Hugging Face), with training script.

3.2 Tokenizer commands (example)

# install
pip install sentencepiece tokenizers

# train sentencepiece
spm_train --input=data/all_texts.txt --model_prefix=sanchari_spm --vocab_size=50000 --model_type=unigram