SozKZ Core: Kazakh Language Models
Collection
Base, instruct, and balanced Kazakh language models trained from scratch — Llama (50M–600M), GPT2, Pythia architectures • 21 items • Updated
Kazakh GPT-2 BPE Tokenizer (50,257 vocab)
A ByteLevel BPE tokenizer trained on a cleaned Kazakh corpus, following the GPT-2 tokenization scheme.
Overview
| Property | Value |
|---|---|
| Vocab size | 50,257 |
| Algorithm | ByteLevel BPE (GPT-2 style) |
| Training data | kazakh-clean-pretrain-text (~78K cleaned documents) |
| Language | Kazakh (kk) |
| License | Apache 2.0 |
Vocabulary composition
- 256 byte-level base tokens
- ~49,634 BPE merges learned from Kazakh text
- 3 special tokens:
<|endoftext|>,<|padding|>,<|startoftext|> - 360 Unicode digit characters (Arabic-Indic, Devanagari, etc.)
- Total: 50,257
Special tokens
| Token | Role |
|---|---|
| `< | endoftext |
| `< | startoftext |
| `< | padding |
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("stukenov/kazakh-gpt2-50k")
text = "Қазақстан — Орталық Азиядағы мемлекет."
tokens = tokenizer.encode(text)
print(f"{len(tokens)} tokens: {tokens}")
print(tokenizer.decode(tokens))
Tokenization examples
| Text | Tokens |
|---|---|
| Қазақстан — Орталық Азиядағы мемлекет. | 6 |
| Бүгін ауа райы жақсы болады. | 6 |
| 2024 жылы халықаралық конференция өтеді. | 7 |
Training details
- Source corpus: 5 domains — OSCAR, Kazakh News, Kazakh Books, Leipzig, CC-100
- Cleaning pipeline: NFC normalization → script profile filter → fastText LID → junk removal → repetition filter → exact + MinHash near-dedup → domain balancing
- min_frequency: 2 (tokens appearing less than twice are excluded from merges)
Intended use
Pre-training Kazakh language models (GPT-2, LLaMA-style, etc.). Optimized for Kazakh Cyrillic script with compact encoding.
Citation
@misc{kazakh-gpt2-50k,
author = {Saken Tukenov},
title = {Kazakh GPT-2 BPE Tokenizer (50K vocab)},
year = {2026},
url = {https://huggingface.co/stukenov/kazakh-gpt2-50k}
}