---
language:
  - kn
  - en
license: cc-by-nc-4.0
library_name: peft
base_model: unsloth/Qwen3-4B-unsloth-bnb-4bit
tags:
  - kannada
  - kanglish
  - romanized-kannada
  - code-mixed
  - transliteration
  - normalization
  - qlora
  - lora
  - unsloth
  - trl
  - peft
  - text-generation
datasets:
  - ai4bharat/Aksharantar
  - ai4bharat/samanantar
pipeline_tag: text-generation
---

# Qwen3 4B Kanglish Normalizer LoRA

A Kannada-English **Kanglish normalizer** fine-tuned from `unsloth/Qwen3-4B` on my local 5070Ti alongside a Ryzen 7 9700x. Trained for only 2.5 hours on 22000 examples.

This model converts informal Kannada written in Roman script, Kannada script, English, or mixed Kannada-English form into structured JSON containing:

* Kannada-script rendering
* Natural normalized Kannada
* Formal Kannada
* English translation
* Token-level language tags

## Example

### Input

```text
ivattu office ge baralla swalpa fever ide
```

### Output

```json
{
  "english_translation": "I cannot come to the office today because I have a slight fever.",
  "formal_kannada": "ಇಂದು ಸ್ವಲ್ಪ ಜ್ವರ ಇರುವುದರಿಂದ ಕಚೇರಿಗೆ ಬರಲು ಸಾಧ್ಯವಿಲ್ಲ.",
  "kannada_script": "ಇವತ್ತು ಆಫೀಸಿಗೆ ಬರಲ್ಲ, ಸ್ವಲ್ಪ ಜ್ವರ ಇದೆ.",
  "normalized_kannada": "ಇವತ್ತು ಆಫೀಸಿಗೆ ಬರಲ್ಲ, ಸ್ವಲ್ಪ ಜ್ವರ ಇದೆ.",
  "tokens": [
    {"lang": "kn_roman", "text": "ivattu"},
    {"lang": "en", "text": "office"},
    {"lang": "kn_roman", "text": "ge"},
    {"lang": "kn_roman", "text": "baralla"},
    {"lang": "kn_roman", "text": "swalpa"},
    {"lang": "en", "text": "fever"},
    {"lang": "kn_roman", "text": "ide"}
  ]
}
```
### More Examples
```
Input - "nanu ellige barabeku"

FT model - 
Output (English) - "I need to be where"
Output (Kannada) - "ನನು ಎಲ್ಲಿಗೆ ಬರಬೇಕು"

Default Model -
Output (English) - "I am here with a barbecue"
Output (Kannada) - ನ"ಾನು ಇಲ್ಲಿ ಬರ್ಬೆಕ್ಯೂ ಇದೆ"
```
## Intended Use

This model is designed for:

* Romanized Kannada to Kannada-script normalization
* Kannada-English code-mixed text cleanup
* Kanglish search-query normalization
* WhatsApp-style Kannada preprocessing
* Kannada dataset cleaning
* Kannada RAG query normalization
* Kannada chatbot input normalization
* Bilingual Kannada-English NLP pipelines

It is **not** intended to be a general-purpose chatbot.

## Output Schema

The model is trained to return JSON only:

```json
{
  "kannada_script": "string",
  "normalized_kannada": "string",
  "formal_kannada": "string",
  "english_translation": "string",
  "tokens": [
    {
      "text": "string",
      "lang": "kn_roman | kn | en | name | number | punctuation | other"
    }
  ]
}
```

## Model Details

* **Base model:** `unsloth/Qwen3-4B`
* **Fine-tuning method:** QLoRA
* **Training framework:** Unsloth + TRL SFTTrainer
* **Adapter type:** LoRA
* **Primary languages:** Kannada, English
* **Task:** Structured Kanglish normalization
* **Context length used during training:** 2048 tokens
* **Number of examples passed:** 22000 examples, 1 epoch

## Training Data

The training data was built from three sources:

### 1. Aksharantar Kannada Transliteration Data

Used for Kannada word-level romanization and transliteration supervision.

This teaches the model mappings such as:

```text
ivattu -> ಇವತ್ತು
nange -> ನನಗೆ
madbeku -> ಮಾಡಬೇಕು
```

### 2. Samanantar Kannada-English Parallel Data

Used to create synthetic sentence-level Kanglish examples.

The pipeline converts Kannada sentences into romanized/noisy Kanglish inputs and uses the original Kannada and English sentence pair as supervision.

### 3. Manual Kanglish Gold Examples

A hand-written JSONL file of high-quality Kanglish examples was added to teach realistic chat-style Kannada, code-mixing, formal rewriting, and practical phrase usage.

Example categories include:

* Office messages
* Travel
* Health
* Payments
* Government forms
* Education
* Family messages
* ML/dev terminology
* Search queries
* Daily conversation

## Dataset Construction

The preprocessing pipeline creates examples of this form:

```json
{
  "source": "manual_gold",
  "messages": [
    {
      "role": "system",
      "content": "You are a Kannada-English Kanglish normalizer..."
    },
    {
      "role": "user",
      "content": "nange ee application submit madbeku"
    },
    {
      "role": "assistant",
      "content": "{\"kannada_script\":\"ನನಗೆ ಈ ಅಪ್ಲಿಕೇಶನ್ ಸಲ್ಲಿಸಬೇಕು.\",...}"
    }
  ]
}
```

The final generated dataset used for the first run contained:

* Train rows: `137979`
* Eval rows: `2000`

## Training Configuration

Initial training configuration:

```yaml
model_name: unsloth/Qwen3-4B
max_seq_length: 2048
load_in_4bit: true
method: qlora
lora_r: 32
lora_alpha: 64
lora_dropout: 0
target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
per_device_train_batch_size: 2
gradient_accumulation_steps: 2
effective_batch_size: 4
learning_rate: 2e-4
lr_scheduler_type: cosine
warmup_ratio: 0.03
optimizer: adamw_8bit
epochs: 1
bf16: true
packing: false
gradient_checkpointing: true
power drawn: approx 230W for 2hrs
Max temperature: 63C
```

## Usage

Install dependencies:

```bash
pip install unsloth transformers peft trl accelerate bitsandbytes
```

Load the model:

```python
import torch
from unsloth import FastLanguageModel

model_name = "RakshithFury/qwen3-4b-kanglish-normalizer-lora"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_name,
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

FastLanguageModel.for_inference(model)
```

Run inference:

```python
SYSTEM_PROMPT = """You are a Kannada-English Kanglish normalizer.
Given informal Kannada written in Roman script, Kannada script, English, or mixed form, return strict JSON with:
- kannada_script: the best Kannada-script rendering of the input
- normalized_kannada: natural Kannada preserving the input's register
- formal_kannada: polite/formal Kannada
- english_translation: faithful English meaning
- tokens: token-level language tags using kn_roman, kn, en, name, number, punctuation, other
Do not add facts. Preserve names, numbers, dates, URLs, and technical terms.
Return JSON only."""

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "ivattu office ge baralla swalpa fever ide"},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=False,
)

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)

print(response)
```

## Recommended Decoding

For deterministic normalization, use greedy decoding:

```python
do_sample=False
temperature=None
top_p=None
```

For production use, validate that the output parses as JSON. If parsing fails, retry once with the same prompt and lower `max_new_tokens`, or use a JSON repair step.


## Limitations

The model may struggle with:

* Rare dialect-specific Roman spellings
* Highly ambiguous Romanized Kannada
* Long passages with multiple speakers
* Complex wordings and slangs
  
## Safety and Responsible Use

This model is a text normalization model. It should not be used as the sole authority for:

* Legal document interpretation
* Medical advice
* Financial decisions
* Government benefit eligibility
* Identity verification
* High-stakes translation

For user-facing systems, show the normalized output for review when accuracy matters.

## License

This model card lists `cc-by-nc-4.0` because the current training pipeline uses Samanantar, whose Hugging Face dataset card lists a non-commercial Creative Commons license.

If you retrain without non-commercial data, update the license accordingly.

## Citation

If you use this model or dataset, please cite the upstream resources used in the pipeline:

```bibtex
@misc{kanglish_normalizer_qwen3_4b_lora,
  title = {Qwen3 4B Kanglish Normalizer LoRA},
  author = {Rakshith Rao},
  year = {2026},
  publisher = {Hugging Face},
  note = {Kannada-English Kanglish normalization model}
}
```