Upload folder using huggingface_hub

README.md

@@ -16,7 +16,7 @@ pipeline_tag: text-generation
 
 # Pothana Base 300M
 
-A **726M parameter** LLaMA-style language model trained **from scratch** on Telugu text.
+A **387M parameter** LLaMA-style language model trained **from scratch** on Telugu text.
 
 Named after [Bammera Pothana](https://en.wikipedia.org/wiki/Bammera_Pothana), the celebrated 15th-century Telugu poet who authored the *Andhra Maha Bhagavatamu*.
 
@@ -28,9 +28,9 @@ Developed by **[Dvitva AI](https://dvitva.ai)**.
 |---|---|
 | **Model** | pothana-base-300M |
 | **Architecture** | LLaMA (RoPE + SwiGLU + RMSNorm + GQA) |
-| **Parameters** | 726M |
+| **Parameters** | 387M (unique) |
 | **Hidden size** | 1024 |
-| **Layers** | 60 |
+| **Layers** | 30 unique (60 effective via weight sharing) |
 | **Attention heads** | 16 Q / 4 KV (Grouped Query Attention) |
 | **Intermediate size** | 2816 |
 | **Context length** | 2048 |
@@ -46,11 +46,13 @@ Developed by **[Dvitva AI](https://dvitva.ai)**.
 ```python
 from transformers import pipeline
 
-pipe = pipeline("text-generation", model="dvitvaai/pothana-base-300M")
+pipe = pipeline("text-generation", model="dvitvaai/pothana-base-300M", trust_remote_code=True)
 result = pipe("తెలుగు భాష", max_new_tokens=50, do_sample=True, temperature=0.8)
 print(result[0]["generated_text"])
 ```
 
+> **Note**: `trust_remote_code=True` is required for the custom tokenizer that cleans up SentencePiece word boundary markers for readable output.
+
 ### Manual loading
 
 ```python
@@ -58,7 +60,7 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
 import torch
 
 model = AutoModelForCausalLM.from_pretrained("dvitvaai/pothana-base-300M")
-tokenizer = AutoTokenizer.from_pretrained("dvitvaai/pothana-base-300M")
+tokenizer = AutoTokenizer.from_pretrained("dvitvaai/pothana-base-300M", trust_remote_code=True)
 
 text = "తెలుగు భాష చాలా అందమైనది"
 inputs = tokenizer(text, return_tensors="pt")
@@ -88,7 +90,7 @@ This model uses a **SentencePiece Unigram** tokenizer with a 48K vocabulary, tra
 
 Key features:
 - **Grouped Query Attention (GQA)**: 16 query heads, 4 KV heads — 4x KV cache reduction
-- **Block-wise Weight Sharing**: 60 HF layers mapped from 30 unique blocks (each used twice), following MobileLLM-LS
+- **Block-wise Weight Sharing**: 30 unique blocks, each used twice = 60 effective layers (MobileLLM-LS)
 - **SwiGLU MLP** with 2816 intermediate size
 - **RoPE** positional encoding (theta=10000.0)
 - **RMSNorm** (no bias in any linear layer)
@@ -106,7 +108,7 @@ Key features:
 
 - This is a **base model** (not instruction-tuned) — it performs text completion, not instruction following
 - Trained primarily on Telugu text; limited multilingual capability
-- Small model size (726M) limits reasoning and knowledge capacity
+- Small model size (387M) limits reasoning and knowledge capacity
 
 ## License
 

tokenizer_class.py

@@ -0,0 +1,19 @@
+"""Custom Telugu SentencePiece tokenizer with clean decode output."""
+from transformers import LlamaTokenizer
+
+
+class TeluguSPTokenizer(LlamaTokenizer):
+    """Telugu SentencePiece tokenizer that produces clean decoded text.
+
+    Extends LlamaTokenizer to clean up SentencePiece's ▁ (U+2581) word
+    boundary markers, producing natural text output.
+    """
+
+    def decode(self, token_ids, skip_special_tokens=False, **kwargs):
+        text = super().decode(token_ids, skip_special_tokens=skip_special_tokens, **kwargs)
+        # SentencePiece uses ▁ (U+2581) as word boundary marker
+        # Replace with space, then clean up
+        text = text.replace("\u2581", " ")
+        # Clean up extra whitespace
+        text = " ".join(text.split())
+        return text

tokenizer_config.json

@@ -1,5 +1,11 @@
 {
   "tokenizer_class": "LlamaTokenizer",
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenizer_class.TeluguSPTokenizer",
+      "tokenizer_class.TeluguSPTokenizer"
+    ]
+  },
   "model_max_length": 2048,
   "bos_token": "<bos>",
   "eos_token": "<eos>",