---
language:
  - te
license: apache-2.0
tags:
  - telugu
  - llama
  - causal-lm
  - sentencepiece
  - gqa
  - weight-sharing
  - from-scratch
library_name: transformers
pipeline_tag: text-generation
---

# Pothana Base 300M

A **387M parameter** LLaMA-style language model trained **from scratch** on Telugu text.

Named after [Bammera Pothana](https://en.wikipedia.org/wiki/Bammera_Pothana), the celebrated 15th-century Telugu poet who authored the *Andhra Maha Bhagavatamu*.

Developed by **[Dvitva AI](https://dvitva.ai)**.

## Model Details

| | |
|---|---|
| **Model** | pothana-base-300M |
| **Architecture** | LLaMA (RoPE + SwiGLU + RMSNorm + GQA) |
| **Parameters** | 387M (unique) |
| **Hidden size** | 1024 |
| **Layers** | 30 unique (60 effective via weight sharing) |
| **Attention heads** | 16 Q / 4 KV (Grouped Query Attention) |
| **Intermediate size** | 2816 |
| **Context length** | 2048 |
| **Vocab size** | 48,000 |
| **Tokenizer** | SentencePiece Unigram (48K) |
| **Training** | Single GPU, bf16 mixed precision |
| **Developed by** | [Dvitva AI](https://dvitva.ai) |

## Quick Start

### Using pipeline

```python
from transformers import pipeline

pipe = pipeline("text-generation", model="dvitvaai/pothana-base-300M", trust_remote_code=True)
result = pipe("తెలుగు భాష", max_new_tokens=50, do_sample=True, temperature=0.8)
print(result[0]["generated_text"])
```

> **Note**: `trust_remote_code=True` is required for the custom tokenizer that cleans up SentencePiece word boundary markers for readable output.

### Manual loading

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("dvitvaai/pothana-base-300M")
tokenizer = AutoTokenizer.from_pretrained("dvitvaai/pothana-base-300M", trust_remote_code=True)

text = "తెలుగు భాష చాలా అందమైనది"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.8,
        top_k=50,
        do_sample=True,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Tokenizer

This model uses a **SentencePiece Unigram** tokenizer with a 48K vocabulary, trained directly on Telugu text.

- Handles raw Telugu text directly (no preprocessing needed)
- Byte-fallback for out-of-vocabulary characters
- Split digits for better number handling
- NFKC normalization

## Architecture

Key features:
- **Grouped Query Attention (GQA)**: 16 query heads, 4 KV heads — 4x KV cache reduction
- **Block-wise Weight Sharing**: 30 unique blocks, each used twice = 60 effective layers (MobileLLM-LS)
- **SwiGLU MLP** with 2816 intermediate size
- **RoPE** positional encoding (theta=10000.0)
- **RMSNorm** (no bias in any linear layer)

## Training

- **Data**: Telugu text corpus (Sangraha dataset)
- **Preprocessing**: SentencePiece tokenization (raw text)
- **Optimizer**: AdamW (lr=3e-4, weight_decay=0.1, beta1=0.9, beta2=0.95)
- **Schedule**: WSD (Warmup-Stable-Decay)
- **Precision**: bf16 mixed precision
- **Hardware**: Single NVIDIA B200 GPU

## Limitations

- This is a **base model** (not instruction-tuned) — it performs text completion, not instruction following
- Trained primarily on Telugu text; limited multilingual capability
- Small model size (387M) limits reasoning and knowledge capacity

## License

Apache 2.0

## Citation

If you use this model, please cite:

```
@misc{pothana-base-300M,
  title={Pothana Base 300M: A Telugu Language Model},
  author={Dvitva AI},
  year={2025},
  url={https://huggingface.co/dvitvaai/pothana-base-300M}
}
```