File size: 2,461 Bytes
5b8355e 6b2eb3b 5b8355e 7d4a402 5b8355e 7d4a402 5b8355e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | ---
license: apache-2.0
language:
- en
library_name: transformers
---
# **ArlowGPT Tokenizer**
### Overview
The **ArlowGPT Tokenizer** is a byte pair encoding (BPE) tokenizer developed from scratch, optimized for large-scale language modeling and text generation tasks. It features a vocabulary size of **59,575 tokens** and supports a maximum context length of **131,072 tokens**, making it suitable for handling extremely long documents and sequences.
### Key Features
- **Vocabulary Size**: 59,575 tokens
- **Maximum Context Length**: 131,072 tokens
- **Tokenizer Type**: Byte Pair Encoding (BPE)
- **Special Tokens**:
- `<pad>`: Padding token used for sequence alignment.
- `<mask>`: Special token for masked language modeling tasks.
- `<eos>`: End-of-sequence token.
- `<bos>`: Beginning-of-sequence token.
- **Trained From Scratch**: The tokenizer was trained from scratch using a large corpus of English and multilingual text.
### Training Data
The tokenizer was trained on **Wikipedia**, ensuring high coverage of general knowledge and domain-specific terms. Although primarily optimized for English, it also includes some multilingual capability due to the nature of the training dataset.
### Intended Use Cases
This tokenizer is designed for **general-purpose language modeling** and is suitable for tasks such as:
- Autoregressive text generation
- Long-context summarization
- Conversational AI
- Information retrieval over large documents
- General NLP tasks requiring long context processing
### Supported Languages
- **Primary Language**: English
- **Secondary Support**: Some multilingual content
### Performance & Benchmarks
No formal benchmarks have been conducted yet, but the tokenizer has been designed for efficiency in both tokenization speed and memory usage, with a focus on handling extremely long contexts up to **131,072 tokens**.
### Limitations
- **Multilingual Coverage**: While the tokenizer includes some multilingual tokens, it is primarily optimized for English text, and performance on non-English languages may vary.
- **No Benchmarked Metrics**: The tokenizer has not undergone formal benchmarking for speed or performance across various tasks.
### Citation
If you use the **ArlowGPT Tokenizer** in your work, please cite it as:
```
@misc{arlowgpt_tokenizer,
title={ArlowGPT Tokenizer},
author={yuchenxie},
year={2025},
howpublished={\url{https://huggingface.co/yuchenxie/ArlowGPT-Tokenizer}}
}
``` |