| # Sanchari — Technical Blueprint (Full — Investor Grade) |
|
|
| > Version: v0.1 |
| > Purpose: Complete technical blueprint to develop Sanchari-S → Sanchari-M → Sanchari-L |
| > Target audience: engineers, infrastructure teams, investors |
|
|
| --- |
|
|
| ## Summary (one line) |
| Build a practical, India-focused multilingual instruction-following LM family (S: ~200–350M, M: ~1–3B, L: 7B+) using modern efficient training (PyTorch + DeepSpeed/Accelerate + FlashAttention), with explicit data provenance, safety audits, and production deployment targets. |
|
|
| --- |
|
|
| ## 1. Design principles (what we deliver and why) |
| 1. **Practicality first** — Sanchari-S must be cheap to train and fast to run for API/mobile use. |
| 2. **Indian language competence** — prioritize English (Indian), Hindi, Telugu, mixed-script. |
| 3. **Safety & governance** — every training stage includes PII scrubbing and red-team testing before any checkpoint release. |
| 4. **Modular pipeline** — tokenizer, preprocessing, training config, adapter-based instruction tuning. |
| 5. **Deliverables at each phase** — private checkpoints for investors (under NDA), evaluation reports, HF repo + demo. |
|
|
| --- |
|
|
| ## 2. Model architecture & approach |
|
|
| ### 2.1 Base model family |
| - **Sanchari-S:** ~200–350M parameters. Decoder-only transformer (GPT-like). Primary use: fast inference, API & mobile clients. |
| - **Sanchari-M:** ~1–3B parameters. Better instruction following and multi-turn coherence. |
| - **Sanchari-L:** ~7B+ parameters. Full foundation for enterprise applications. |
|
|
| ### 2.2 Model topology (recommended) |
| - **Type:** Decoder-only transformer. |
| - **Layer scaling (example recipes):** |
| - Sanchari-S: 24 layers × 8 heads × 2048 hidden (example ≈ 300M) |
| - Sanchari-M: 36 layers × 16 heads × 4096 hidden (example ≈ 1.3B) |
| - Sanchari-L: 48–80 layers × 32 heads × 6144 hidden (approx 7B) |
| - **Attention:** FlashAttention2 compatible (for GPU memory and speed). |
| - **Norms and embeddings:** RMSNorm or LayerNorm, rotary positional embeddings (RoPE) for stable long-range. |
| - **Efficiency:** Provide LoRA adapters and QLoRA options for cost-effective instruction tuning. |
|
|
| > Rationale: decoder-only is industry standard for instruction-following and easier to deploy as an API. |
|
|
| --- |
|
|
| ## 3. Tokenizer & text processing |
| ### 3.1 Tokenizer strategy |
| - **Tokenizer:** SentencePiece Unigram or BPE with ~50k vocabulary, trained on mixed Indic + English corpus. |
| - **Features required:** |
| - Mixed script normalization (normalize Unicode, NFKC) |
| - Preserve whitespace tokens for code-like text |
| - Subword segmentation for Indic scripts |
| - **Tooling:** `sentencepiece` or `tokenizers` (Hugging Face), with training script. |
|
|
| ### 3.2 Tokenizer commands (example) |
| ```bash |
| # install |
| pip install sentencepiece tokenizers |
| |
| # train sentencepiece |
| spm_train --input=data/all_texts.txt --model_prefix=sanchari_spm --vocab_size=50000 --model_type=unigram |
