---
language:
- vi
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- tokenizer
- byte-level-bpe
- vietnamese
- english
- code
- qwen-style
license: apache-2.0
---

# Vietnamese-Tokenizer

`Vietnamese-Tokenizer` is a **48,000-token Byte-level BPE tokenizer** designed primarily for Vietnamese language models trained from scratch.

The tokenizer is optimized for Vietnamese while retaining practical coverage of English and source code. It is intended to be architecture-independent and can be used with Qwen-style, LLaMA-style, or other autoregressive language model architectures as long as the model configuration uses the same vocabulary and token IDs.

## Key Features

- **Vocabulary size:** 48,000
- **Algorithm:** Byte-level BPE
- **Unicode normalization:** NFC
- **Lowercasing:** No
- **Unknown token:** None
- **Fast tokenizer:** Yes
- **Byte-level coverage:** Any UTF-8 text can be represented
- **Vietnamese-focused vocabulary**
- Includes English and source-code coverage
- Includes reserved token IDs for future extensions

## Special Tokens

| Token | ID |
|---|---:|
| `<|pad|>` | 0 |
| `<|bos|>` | 1 |
| `<|eos|>` | 2 |
| `<|im_start|>` | 3 |
| `<|im_end|>` | 4 |
| `<|fim_prefix|>` | 5 |
| `<|fim_middle|>` | 6 |
| `<|fim_suffix|>` | 7 |

The tokenizer also reserves IDs `8..263` as:

```text
<|reserved_0|>
...
<|reserved_255|>
```

These reserved tokens are intentionally kept stable so future model variants can introduce additional control tokens without changing the existing token-to-ID mapping.

## Training Corpus

The tokenizer was trained on approximately **8 GiB** of mixed-domain text.

Approximate composition:

| Domain | Share |
|---|---:|
| Vietnamese news | 70% |
| Vietnamese Wikipedia | 15% |
| English general text | 12% |
| Source code | 3% |

The source-code portion includes multiple programming languages, with higher weight assigned to commonly used languages such as Python, JavaScript, Java, C++, and Go.

The corpus used to train this tokenizer is published separately.

## Corpus Statistics

The tokenizer training corpus contained approximately:

- **1,687,254** Vietnamese news documents
- **1,035,370** Vietnamese Wikipedia documents
- **326,223** English documents
- **~107K** code documents across multiple programming languages

The total serialized corpus size was approximately **8 GiB**.

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "PaxiAI/Vietnamese-Tokenizer",
    use_fast=True,
)

text = "Trí tuệ nhân tạo đang thay đổi cách con người làm việc."

ids = tokenizer.encode(
    text,
    add_special_tokens=False,
)

print(ids)
print(tokenizer.decode(ids))
```

## Chat Template

The tokenizer includes a simple ChatML-style template:

```text
<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
Xin chào!
<|im_end|>
<|im_start|>assistant
Chào bạn!
<|im_end|>
```

Example:

```python
messages = [
    {
        "role": "system",
        "content": "Bạn là một trợ lý hữu ích."
    },
    {
        "role": "user",
        "content": "Xin chào!"
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

print(prompt)
```

## Unicode Handling

The tokenizer applies **Unicode NFC normalization**.

For example, canonically equivalent NFC and NFD forms of Vietnamese text are normalized to the same representation before tokenization.

This is particularly important for Vietnamese because accented characters may otherwise appear in multiple Unicode representations.

## Validation

Before release, the tokenizer passed a production validation suite covering:

- fixed-text encode/decode round trips
- Vietnamese NFC/NFD normalization
- special-token ID stability
- tokenizer save/reload consistency
- chat-template rendering
- randomized Unicode fuzz testing
- no unknown-token output
- long-input tokenization
- fast-tokenizer loading
- exact vocabulary-size validation

A fuzz test of **10,000 randomly generated Unicode strings** completed with zero failures.

The tokenizer also produced zero unknown tokens during validation.

## Performance Notes

On manual Vietnamese examples, the tokenizer typically produced roughly **3.6-4.4 characters per token**, depending on the sentence.

Example:

```text
Nguyễn Thị Kim Hương đang được cấp cứu tại Bệnh viện Chợ Rẫy.
```

was tokenized into approximately one token per common Vietnamese syllable or word component.

The tokenizer is intentionally optimized more strongly for Vietnamese than for code identifiers or rare English terms.

## Compatibility Warning

Once a model has been pretrained with this tokenizer, the following must remain unchanged:

- vocabulary
- token-to-ID mapping
- normalization rules
- special-token IDs

Changing any of these creates a different tokenizer and should be released under a new version.

## Intended Use

This tokenizer is suitable for:

- Vietnamese language model pretraining
- bilingual Vietnamese-English language models
- Vietnamese instruction-tuned models
- small and medium autoregressive language models
- experimental Qwen-style or LLaMA-style architectures
- models with limited parameter budgets where a very large vocabulary would be inefficient

## Limitations

- The training corpus is heavily weighted toward Vietnamese news and Wikipedia.
- Conversational Vietnamese is less represented than formal written Vietnamese.
- Code represents only a small portion of the tokenizer training data.
- The tokenizer is not intended to be optimal for multilingual models covering many languages equally.
- A tokenizer alone does not determine model quality; pretraining data and training methodology remain critical.

## License

This repository contains the tokenizer artifact itself.

Please ensure that the selected repository license is compatible with the licensing and redistribution requirements of the tokenizer artifact and its training-data sources.