| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| --- |
| |
| # **ArlowGPT Tokenizer** |
|
|
| ### Overview |
| The **ArlowGPT Tokenizer** is a byte pair encoding (BPE) tokenizer developed from scratch, optimized for large-scale language modeling and text generation tasks. It features a vocabulary size of **59,575 tokens** and supports a maximum context length of **131,072 tokens**, making it suitable for handling extremely long documents and sequences. |
|
|
| ### Key Features |
| - **Vocabulary Size**: 59,575 tokens |
| - **Maximum Context Length**: 131,072 tokens |
| - **Tokenizer Type**: Byte Pair Encoding (BPE) |
| - **Special Tokens**: |
| - `<pad>`: Padding token used for sequence alignment. |
| - `<mask>`: Special token for masked language modeling tasks. |
| - `<eos>`: End-of-sequence token. |
| - `<bos>`: Beginning-of-sequence token. |
| - **Trained From Scratch**: The tokenizer was trained from scratch using a large corpus of English and multilingual text. |
|
|
| ### Training Data |
| The tokenizer was trained on **Wikipedia**, ensuring high coverage of general knowledge and domain-specific terms. Although primarily optimized for English, it also includes some multilingual capability due to the nature of the training dataset. |
|
|
| ### Intended Use Cases |
| This tokenizer is designed for **general-purpose language modeling** and is suitable for tasks such as: |
| - Autoregressive text generation |
| - Long-context summarization |
| - Conversational AI |
| - Information retrieval over large documents |
| - General NLP tasks requiring long context processing |
|
|
| ### Supported Languages |
| - **Primary Language**: English |
| - **Secondary Support**: Some multilingual content |
|
|
| ### Performance & Benchmarks |
| No formal benchmarks have been conducted yet, but the tokenizer has been designed for efficiency in both tokenization speed and memory usage, with a focus on handling extremely long contexts up to **131,072 tokens**. |
|
|
| ### Limitations |
| - **Multilingual Coverage**: While the tokenizer includes some multilingual tokens, it is primarily optimized for English text, and performance on non-English languages may vary. |
| - **No Benchmarked Metrics**: The tokenizer has not undergone formal benchmarking for speed or performance across various tasks. |
|
|
| ### Citation |
| If you use the **ArlowGPT Tokenizer** in your work, please cite it as: |
| ``` |
| @misc{arlowgpt_tokenizer, |
| title={ArlowGPT Tokenizer}, |
| author={yuchenxie}, |
| year={2025}, |
| howpublished={\url{https://huggingface.co/yuchenxie/ArlowGPT-Tokenizer}} |
| } |
| ``` |
