Hugging Face's logo

yuchenxie
/
ArlowGPT-Tokenizer

Transformers
English
Model card Files
xet
 Community
yuchenxie commited on
Commit
7d4a402
·
verified ·
1 Parent(s): 94744d4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -8,10 +8,10 @@ library_name: transformers
8
  # **ArlowGPT Tokenizer**
9
 
10
  ### Overview
11
- The **ArlowGPT Tokenizer** is a byte pair encoding (BPE) tokenizer developed from scratch, optimized for large-scale language modeling and text generation tasks. It features a vocabulary size of **131,072 tokens** and supports a maximum context length of **131,072 tokens**, making it suitable for handling extremely long documents and sequences.
12
 
13
  ### Key Features
14
- - **Vocabulary Size**: 131,072 tokens
15
  - **Maximum Context Length**: 131,072 tokens
16
  - **Tokenizer Type**: Byte Pair Encoding (BPE)
17
  - **Special Tokens**:
 
8
  # **ArlowGPT Tokenizer**
9
 
10
  ### Overview
11
+ The **ArlowGPT Tokenizer** is a byte pair encoding (BPE) tokenizer developed from scratch, optimized for large-scale language modeling and text generation tasks. It features a vocabulary size of **59,575 tokens** and supports a maximum context length of **131,072 tokens**, making it suitable for handling extremely long documents and sequences.
12
 
13
  ### Key Features
14
+ - **Vocabulary Size**: 59,575 tokens
15
  - **Maximum Context Length**: 131,072 tokens
16
  - **Tokenizer Type**: Byte Pair Encoding (BPE)
17
  - **Special Tokens**: