Upload folder using huggingface_hub

README.md

@@ -1,3 +1,276 @@
 ---
-license: mit
+language:
+- vi
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- tokenizer
+- byte-level-bpe
+- vietnamese
+- english
+- code
+- qwen-style
+license: apache-2.0
 ---
+
+# VietToken
+
+`VietToken` is a **48,000-token Byte-level BPE tokenizer** designed primarily for Vietnamese language models trained from scratch.
+
+The tokenizer is optimized for Vietnamese while retaining practical coverage of English and source code. It is intended to be architecture-independent and can be used with Qwen-style, LLaMA-style, or other autoregressive language model architectures as long as the model configuration uses the same vocabulary and token IDs.
+
+## Key Features
+
+- **Vocabulary size:** 48,000
+- **Algorithm:** Byte-level BPE
+- **Unicode normalization:** NFC
+- **Lowercasing:** No
+- **Unknown token:** None
+- **Fast tokenizer:** Yes
+- **Byte-level coverage:** Any UTF-8 text can be represented
+- **Vietnamese-focused vocabulary**
+- Includes English and source-code coverage
+- Includes reserved token IDs for future extensions
+
+## Special Tokens
+
+| Token | ID |
+|---|---:|
+| `<|pad|>` | 0 |
+| `<|bos|>` | 1 |
+| `<|eos|>` | 2 |
+| `<|im_start|>` | 3 |
+| `<|im_end|>` | 4 |
+| `<|fim_prefix|>` | 5 |
+| `<|fim_middle|>` | 6 |
+| `<|fim_suffix|>` | 7 |
+
+The tokenizer also reserves IDs `8..263` as:
+
+```text
+<|reserved_0|>
+...
+<|reserved_255|>
+```
+
+These reserved tokens are intentionally kept stable so future model variants can introduce additional control tokens without changing the existing token-to-ID mapping.
+
+## Training Corpus
+
+The tokenizer was trained on approximately **8 GiB** of mixed-domain text.
+
+Approximate composition:
+
+| Domain | Share |
+|---|---:|
+| Vietnamese news | 70% |
+| Vietnamese Wikipedia | 15% |
+| English general text | 12% |
+| Source code | 3% |
+
+The source-code portion includes multiple programming languages, with higher weight assigned to commonly used languages such as Python, JavaScript, Java, C++, and Go.
+
+The corpus used to train this tokenizer is published separately.
+
+## Corpus Statistics
+
+The tokenizer training corpus contained approximately:
+
+- **1,687,254** Vietnamese news documents
+- **1,035,370** Vietnamese Wikipedia documents
+- **326,223** English documents
+- **~107K** code documents across multiple programming languages
+
+The total serialized corpus size was approximately **8 GiB**.
+
+## Usage
+
+```python
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained(
+    "PaxiAI/VietToken",
+    use_fast=True,
+)
+
+text = "Trí tuệ nhân tạo đang thay đổi cách con người làm việc."
+
+ids = tokenizer.encode(
+    text,
+    add_special_tokens=False,
+)
+
+print(ids)
+print(tokenizer.decode(ids))
+```
+
+Example behavior:
+
+```text
+Input:
+Trí tuệ nhân tạo đang thay đổi cách con người làm việc và học tập.
+
+Decoded:
+Trí tuệ nhân tạo đang thay đổi cách con người làm việc và học tập.
+```
+
+## Chat Template
+
+The tokenizer includes a simple ChatML-style template:
+
+```text
+<|im_start|>system
+You are a helpful assistant.
+<|im_end|>
+<|im_start|>user
+Xin chào!
+<|im_end|>
+<|im_start|>assistant
+Chào bạn!
+<|im_end|>
+```
+
+Example:
+
+```python
+messages = [
+    {
+        "role": "system",
+        "content": "Bạn là một trợ lý hữu ích."
+    },
+    {
+        "role": "user",
+        "content": "Xin chào!"
+    }
+]
+
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+
+print(prompt)
+```
+
+## Unicode Handling
+
+The tokenizer applies **Unicode NFC normalization**.
+
+For example, canonically equivalent NFC and NFD forms of Vietnamese text are normalized to the same representation before tokenization.
+
+This is particularly important for Vietnamese because accented characters may otherwise appear in multiple Unicode representations.
+
+## Validation
+
+Before release, the tokenizer passed a production validation suite covering:
+
+- fixed-text encode/decode round trips
+- Vietnamese NFC/NFD normalization
+- special-token ID stability
+- tokenizer save/reload consistency
+- chat-template rendering
+- randomized Unicode fuzz testing
+- no unknown-token output
+- long-input tokenization
+- fast-tokenizer loading
+- exact vocabulary-size validation
+
+A fuzz test of **10,000 randomly generated Unicode strings** completed with zero failures.
+
+The tokenizer also produced zero unknown tokens during validation.
+
+## Performance Notes
+
+On manual Vietnamese examples, the tokenizer typically produced roughly **3.6-4.4 characters per token**, depending on the sentence.
+
+Example:
+
+```text
+Nguyễn Thị Kim Hương đang được cấp cứu tại Bệnh viện Chợ Rẫy.
+```
+
+was tokenized into approximately one token per common Vietnamese syllable or word component.
+
+The tokenizer is intentionally optimized more strongly for Vietnamese than for code identifiers or rare English terms.
+
+## Architecture Independence
+
+This tokenizer is not tied to a specific model implementation.
+
+It can be used with a model initialized from scratch, for example:
+
+```python
+from transformers import AutoTokenizer, Qwen2Config, Qwen2ForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained(
+    "PaxiAI/VietToken"
+)
+
+config = Qwen2Config(
+    vocab_size=len(tokenizer),
+    hidden_size=1024,
+    intermediate_size=2816,
+    num_hidden_layers=20,
+    num_attention_heads=16,
+    num_key_value_heads=4,
+    pad_token_id=tokenizer.pad_token_id,
+    bos_token_id=tokenizer.bos_token_id,
+    eos_token_id=tokenizer.eos_token_id,
+)
+
+model = Qwen2ForCausalLM(config)
+```
+
+The model above is randomly initialized. No Qwen weights are required.
+
+## Compatibility Warning
+
+Once a model has been pretrained with this tokenizer, the following must remain unchanged:
+
+- vocabulary
+- token-to-ID mapping
+- normalization rules
+- special-token IDs
+
+Changing any of these creates a different tokenizer and should be released under a new version.
+
+## Recommended Versioning
+
+This release should be treated as:
+
+```text
+VietnameseTokenizer-v1
+```
+
+Future incompatible tokenizer changes should use a new version such as:
+
+```text
+VietnameseTokenizer-v2
+```
+
+## Intended Use
+
+This tokenizer is suitable for:
+
+- Vietnamese language model pretraining
+- bilingual Vietnamese-English language models
+- Vietnamese instruction-tuned models
+- small and medium autoregressive language models
+- experimental Qwen-style or LLaMA-style architectures
+- models with limited parameter budgets where a very large vocabulary would be inefficient
+
+## Limitations
+
+- The training corpus is heavily weighted toward Vietnamese news and Wikipedia.
+- Conversational Vietnamese is less represented than formal written Vietnamese.
+- Code represents only a small portion of the tokenizer training data.
+- The tokenizer is not intended to be optimal for multilingual models covering many languages equally.
+- A tokenizer alone does not determine model quality; pretraining data and training methodology remain critical.
+
+## License
+
+This repository contains the tokenizer artifact itself.
+
+Please ensure that the selected repository license is compatible with the licensing and redistribution requirements of the tokenizer artifact and its training-data sources.

chat_template.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{{ '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>\n' }}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}

special_tokens_map.json

@@ -0,0 +1,268 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|reserved_0|>",
+    "<|reserved_1|>",
+    "<|reserved_2|>",
+    "<|reserved_3|>",
+    "<|reserved_4|>",
+    "<|reserved_5|>",
+    "<|reserved_6|>",
+    "<|reserved_7|>",
+    "<|reserved_8|>",
+    "<|reserved_9|>",
+    "<|reserved_10|>",
+    "<|reserved_11|>",
+    "<|reserved_12|>",
+    "<|reserved_13|>",
+    "<|reserved_14|>",
+    "<|reserved_15|>",
+    "<|reserved_16|>",
+    "<|reserved_17|>",
+    "<|reserved_18|>",
+    "<|reserved_19|>",
+    "<|reserved_20|>",
+    "<|reserved_21|>",
+    "<|reserved_22|>",
+    "<|reserved_23|>",
+    "<|reserved_24|>",
+    "<|reserved_25|>",
+    "<|reserved_26|>",
+    "<|reserved_27|>",
+    "<|reserved_28|>",
+    "<|reserved_29|>",
+    "<|reserved_30|>",
+    "<|reserved_31|>",
+    "<|reserved_32|>",
+    "<|reserved_33|>",
+    "<|reserved_34|>",
+    "<|reserved_35|>",
+    "<|reserved_36|>",
+    "<|reserved_37|>",
+    "<|reserved_38|>",
+    "<|reserved_39|>",
+    "<|reserved_40|>",
+    "<|reserved_41|>",
+    "<|reserved_42|>",
+    "<|reserved_43|>",
+    "<|reserved_44|>",
+    "<|reserved_45|>",
+    "<|reserved_46|>",
+    "<|reserved_47|>",
+    "<|reserved_48|>",
+    "<|reserved_49|>",
+    "<|reserved_50|>",
+    "<|reserved_51|>",
+    "<|reserved_52|>",
+    "<|reserved_53|>",
+    "<|reserved_54|>",
+    "<|reserved_55|>",
+    "<|reserved_56|>",
+    "<|reserved_57|>",
+    "<|reserved_58|>",
+    "<|reserved_59|>",
+    "<|reserved_60|>",
+    "<|reserved_61|>",
+    "<|reserved_62|>",
+    "<|reserved_63|>",
+    "<|reserved_64|>",
+    "<|reserved_65|>",
+    "<|reserved_66|>",
+    "<|reserved_67|>",
+    "<|reserved_68|>",
+    "<|reserved_69|>",
+    "<|reserved_70|>",
+    "<|reserved_71|>",
+    "<|reserved_72|>",
+    "<|reserved_73|>",
+    "<|reserved_74|>",
+    "<|reserved_75|>",
+    "<|reserved_76|>",
+    "<|reserved_77|>",
+    "<|reserved_78|>",
+    "<|reserved_79|>",
+    "<|reserved_80|>",
+    "<|reserved_81|>",
+    "<|reserved_82|>",
+    "<|reserved_83|>",
+    "<|reserved_84|>",
+    "<|reserved_85|>",
+    "<|reserved_86|>",
+    "<|reserved_87|>",
+    "<|reserved_88|>",
+    "<|reserved_89|>",
+    "<|reserved_90|>",
+    "<|reserved_91|>",
+    "<|reserved_92|>",
+    "<|reserved_93|>",
+    "<|reserved_94|>",
+    "<|reserved_95|>",
+    "<|reserved_96|>",
+    "<|reserved_97|>",
+    "<|reserved_98|>",
+    "<|reserved_99|>",
+    "<|reserved_100|>",
+    "<|reserved_101|>",
+    "<|reserved_102|>",
+    "<|reserved_103|>",
+    "<|reserved_104|>",
+    "<|reserved_105|>",
+    "<|reserved_106|>",
+    "<|reserved_107|>",
+    "<|reserved_108|>",
+    "<|reserved_109|>",
+    "<|reserved_110|>",
+    "<|reserved_111|>",
+    "<|reserved_112|>",
+    "<|reserved_113|>",
+    "<|reserved_114|>",
+    "<|reserved_115|>",
+    "<|reserved_116|>",
+    "<|reserved_117|>",
+    "<|reserved_118|>",
+    "<|reserved_119|>",
+    "<|reserved_120|>",
+    "<|reserved_121|>",
+    "<|reserved_122|>",
+    "<|reserved_123|>",
+    "<|reserved_124|>",
+    "<|reserved_125|>",
+    "<|reserved_126|>",
+    "<|reserved_127|>",
+    "<|reserved_128|>",
+    "<|reserved_129|>",
+    "<|reserved_130|>",
+    "<|reserved_131|>",
+    "<|reserved_132|>",
+    "<|reserved_133|>",
+    "<|reserved_134|>",
+    "<|reserved_135|>",
+    "<|reserved_136|>",
+    "<|reserved_137|>",
+    "<|reserved_138|>",
+    "<|reserved_139|>",
+    "<|reserved_140|>",
+    "<|reserved_141|>",
+    "<|reserved_142|>",
+    "<|reserved_143|>",
+    "<|reserved_144|>",
+    "<|reserved_145|>",
+    "<|reserved_146|>",
+    "<|reserved_147|>",
+    "<|reserved_148|>",
+    "<|reserved_149|>",
+    "<|reserved_150|>",
+    "<|reserved_151|>",
+    "<|reserved_152|>",
+    "<|reserved_153|>",
+    "<|reserved_154|>",
+    "<|reserved_155|>",
+    "<|reserved_156|>",
+    "<|reserved_157|>",
+    "<|reserved_158|>",
+    "<|reserved_159|>",
+    "<|reserved_160|>",
+    "<|reserved_161|>",
+    "<|reserved_162|>",
+    "<|reserved_163|>",
+    "<|reserved_164|>",
+    "<|reserved_165|>",
+    "<|reserved_166|>",
+    "<|reserved_167|>",
+    "<|reserved_168|>",
+    "<|reserved_169|>",
+    "<|reserved_170|>",
+    "<|reserved_171|>",
+    "<|reserved_172|>",
+    "<|reserved_173|>",
+    "<|reserved_174|>",
+    "<|reserved_175|>",
+    "<|reserved_176|>",
+    "<|reserved_177|>",
+    "<|reserved_178|>",
+    "<|reserved_179|>",
+    "<|reserved_180|>",
+    "<|reserved_181|>",
+    "<|reserved_182|>",
+    "<|reserved_183|>",
+    "<|reserved_184|>",
+    "<|reserved_185|>",
+    "<|reserved_186|>",
+    "<|reserved_187|>",
+    "<|reserved_188|>",
+    "<|reserved_189|>",
+    "<|reserved_190|>",
+    "<|reserved_191|>",
+    "<|reserved_192|>",
+    "<|reserved_193|>",
+    "<|reserved_194|>",
+    "<|reserved_195|>",
+    "<|reserved_196|>",
+    "<|reserved_197|>",
+    "<|reserved_198|>",
+    "<|reserved_199|>",
+    "<|reserved_200|>",
+    "<|reserved_201|>",
+    "<|reserved_202|>",
+    "<|reserved_203|>",
+    "<|reserved_204|>",
+    "<|reserved_205|>",
+    "<|reserved_206|>",
+    "<|reserved_207|>",
+    "<|reserved_208|>",
+    "<|reserved_209|>",
+    "<|reserved_210|>",
+    "<|reserved_211|>",
+    "<|reserved_212|>",
+    "<|reserved_213|>",
+    "<|reserved_214|>",
+    "<|reserved_215|>",
+    "<|reserved_216|>",
+    "<|reserved_217|>",
+    "<|reserved_218|>",
+    "<|reserved_219|>",
+    "<|reserved_220|>",
+    "<|reserved_221|>",
+    "<|reserved_222|>",
+    "<|reserved_223|>",
+    "<|reserved_224|>",
+    "<|reserved_225|>",
+    "<|reserved_226|>",
+    "<|reserved_227|>",
+    "<|reserved_228|>",
+    "<|reserved_229|>",
+    "<|reserved_230|>",
+    "<|reserved_231|>",
+    "<|reserved_232|>",
+    "<|reserved_233|>",
+    "<|reserved_234|>",
+    "<|reserved_235|>",
+    "<|reserved_236|>",
+    "<|reserved_237|>",
+    "<|reserved_238|>",
+    "<|reserved_239|>",
+    "<|reserved_240|>",
+    "<|reserved_241|>",
+    "<|reserved_242|>",
+    "<|reserved_243|>",
+    "<|reserved_244|>",
+    "<|reserved_245|>",
+    "<|reserved_246|>",
+    "<|reserved_247|>",
+    "<|reserved_248|>",
+    "<|reserved_249|>",
+    "<|reserved_250|>",
+    "<|reserved_251|>",
+    "<|reserved_252|>",
+    "<|reserved_253|>",
+    "<|reserved_254|>",
+    "<|reserved_255|>"
+  ],
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "pad_token": "<|pad|>"
+}

tokenizer_config.json

@@ -0,0 +1,2386 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|bos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|eos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<|reserved_0|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<|reserved_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<|reserved_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<|reserved_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<|reserved_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<|reserved_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<|reserved_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<|reserved_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<|reserved_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<|reserved_9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<|reserved_10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "19": {
+      "content": "<|reserved_11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "20": {
+      "content": "<|reserved_12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "21": {
+      "content": "<|reserved_13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "22": {
+      "content": "<|reserved_14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "23": {
+      "content": "<|reserved_15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "24": {
+      "content": "<|reserved_16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25": {
+      "content": "<|reserved_17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "26": {
+      "content": "<|reserved_18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "27": {
+      "content": "<|reserved_19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "28": {
+      "content": "<|reserved_20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "29": {
+      "content": "<|reserved_21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30": {
+      "content": "<|reserved_22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "31": {
+      "content": "<|reserved_23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32": {
+      "content": "<|reserved_24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "33": {
+      "content": "<|reserved_25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "34": {
+      "content": "<|reserved_26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "35": {
+      "content": "<|reserved_27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "36": {
+      "content": "<|reserved_28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "37": {
+      "content": "<|reserved_29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "38": {
+      "content": "<|reserved_30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "39": {
+      "content": "<|reserved_31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "40": {
+      "content": "<|reserved_32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "41": {
+      "content": "<|reserved_33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "42": {
+      "content": "<|reserved_34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "43": {
+      "content": "<|reserved_35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "44": {
+      "content": "<|reserved_36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "45": {
+      "content": "<|reserved_37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "46": {
+      "content": "<|reserved_38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "47": {
+      "content": "<|reserved_39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "48": {
+      "content": "<|reserved_40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49": {
+      "content": "<|reserved_41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50": {
+      "content": "<|reserved_42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "51": {
+      "content": "<|reserved_43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "52": {
+      "content": "<|reserved_44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "53": {
+      "content": "<|reserved_45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "54": {
+      "content": "<|reserved_46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "55": {
+      "content": "<|reserved_47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "56": {
+      "content": "<|reserved_48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "57": {
+      "content": "<|reserved_49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "58": {
+      "content": "<|reserved_50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "59": {
+      "content": "<|reserved_51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "60": {
+      "content": "<|reserved_52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "61": {
+      "content": "<|reserved_53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "62": {
+      "content": "<|reserved_54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "63": {
+      "content": "<|reserved_55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64": {
+      "content": "<|reserved_56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65": {
+      "content": "<|reserved_57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "66": {
+      "content": "<|reserved_58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "67": {
+      "content": "<|reserved_59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "68": {
+      "content": "<|reserved_60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "69": {
+      "content": "<|reserved_61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "70": {
+      "content": "<|reserved_62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "71": {
+      "content": "<|reserved_63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "72": {
+      "content": "<|reserved_64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73": {
+      "content": "<|reserved_65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "74": {
+      "content": "<|reserved_66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "75": {
+      "content": "<|reserved_67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "76": {
+      "content": "<|reserved_68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "77": {
+      "content": "<|reserved_69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "78": {
+      "content": "<|reserved_70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "79": {
+      "content": "<|reserved_71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "80": {
+      "content": "<|reserved_72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "81": {
+      "content": "<|reserved_73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "82": {
+      "content": "<|reserved_74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "83": {
+      "content": "<|reserved_75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "84": {
+      "content": "<|reserved_76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "85": {
+      "content": "<|reserved_77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "86": {
+      "content": "<|reserved_78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "87": {
+      "content": "<|reserved_79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "88": {
+      "content": "<|reserved_80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "89": {
+      "content": "<|reserved_81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "90": {
+      "content": "<|reserved_82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "91": {
+      "content": "<|reserved_83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "92": {
+      "content": "<|reserved_84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "93": {
+      "content": "<|reserved_85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "94": {
+      "content": "<|reserved_86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "95": {
+      "content": "<|reserved_87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "96": {
+      "content": "<|reserved_88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "97": {
+      "content": "<|reserved_89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "98": {
+      "content": "<|reserved_90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "99": {
+      "content": "<|reserved_91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "<|reserved_92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "<|reserved_93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "<|reserved_94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "<|reserved_95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "104": {
+      "content": "<|reserved_96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "105": {
+      "content": "<|reserved_97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "106": {
+      "content": "<|reserved_98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "107": {
+      "content": "<|reserved_99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "108": {
+      "content": "<|reserved_100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "109": {
+      "content": "<|reserved_101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110": {
+      "content": "<|reserved_102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "111": {
+      "content": "<|reserved_103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "112": {
+      "content": "<|reserved_104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "113": {
+      "content": "<|reserved_105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "114": {
+      "content": "<|reserved_106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "115": {
+      "content": "<|reserved_107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "116": {
+      "content": "<|reserved_108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "117": {
+      "content": "<|reserved_109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "118": {
+      "content": "<|reserved_110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "119": {
+      "content": "<|reserved_111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "120": {
+      "content": "<|reserved_112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "121": {
+      "content": "<|reserved_113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "122": {
+      "content": "<|reserved_114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "123": {
+      "content": "<|reserved_115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "124": {
+      "content": "<|reserved_116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "125": {
+      "content": "<|reserved_117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "126": {
+      "content": "<|reserved_118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "127": {
+      "content": "<|reserved_119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128": {
+      "content": "<|reserved_120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "129": {
+      "content": "<|reserved_121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "130": {
+      "content": "<|reserved_122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "131": {
+      "content": "<|reserved_123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "132": {
+      "content": "<|reserved_124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "133": {
+      "content": "<|reserved_125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "134": {
+      "content": "<|reserved_126|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "135": {
+      "content": "<|reserved_127|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "136": {
+      "content": "<|reserved_128|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "137": {
+      "content": "<|reserved_129|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "138": {
+      "content": "<|reserved_130|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "139": {
+      "content": "<|reserved_131|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "140": {
+      "content": "<|reserved_132|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "141": {
+      "content": "<|reserved_133|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "142": {
+      "content": "<|reserved_134|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "143": {
+      "content": "<|reserved_135|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "144": {
+      "content": "<|reserved_136|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "145": {
+      "content": "<|reserved_137|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "146": {
+      "content": "<|reserved_138|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147": {
+      "content": "<|reserved_139|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "148": {
+      "content": "<|reserved_140|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "149": {
+      "content": "<|reserved_141|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "150": {
+      "content": "<|reserved_142|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151": {
+      "content": "<|reserved_143|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "152": {
+      "content": "<|reserved_144|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "153": {
+      "content": "<|reserved_145|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "154": {
+      "content": "<|reserved_146|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "155": {
+      "content": "<|reserved_147|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "156": {
+      "content": "<|reserved_148|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "157": {
+      "content": "<|reserved_149|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "158": {
+      "content": "<|reserved_150|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "159": {
+      "content": "<|reserved_151|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "160": {
+      "content": "<|reserved_152|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "161": {
+      "content": "<|reserved_153|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "162": {
+      "content": "<|reserved_154|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "163": {
+      "content": "<|reserved_155|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "164": {
+      "content": "<|reserved_156|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "165": {
+      "content": "<|reserved_157|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "166": {
+      "content": "<|reserved_158|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "167": {
+      "content": "<|reserved_159|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "168": {
+      "content": "<|reserved_160|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "169": {
+      "content": "<|reserved_161|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "170": {
+      "content": "<|reserved_162|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "171": {
+      "content": "<|reserved_163|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "172": {
+      "content": "<|reserved_164|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "173": {
+      "content": "<|reserved_165|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "174": {
+      "content": "<|reserved_166|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "175": {
+      "content": "<|reserved_167|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "176": {
+      "content": "<|reserved_168|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "177": {
+      "content": "<|reserved_169|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "178": {
+      "content": "<|reserved_170|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "179": {
+      "content": "<|reserved_171|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "180": {
+      "content": "<|reserved_172|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "181": {
+      "content": "<|reserved_173|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "182": {
+      "content": "<|reserved_174|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "183": {
+      "content": "<|reserved_175|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "184": {
+      "content": "<|reserved_176|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "185": {
+      "content": "<|reserved_177|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "186": {
+      "content": "<|reserved_178|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "187": {
+      "content": "<|reserved_179|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "188": {
+      "content": "<|reserved_180|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "189": {
+      "content": "<|reserved_181|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "190": {
+      "content": "<|reserved_182|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "191": {
+      "content": "<|reserved_183|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "192": {
+      "content": "<|reserved_184|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "193": {
+      "content": "<|reserved_185|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "194": {
+      "content": "<|reserved_186|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "195": {
+      "content": "<|reserved_187|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "196": {
+      "content": "<|reserved_188|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "197": {
+      "content": "<|reserved_189|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "198": {
+      "content": "<|reserved_190|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "199": {
+      "content": "<|reserved_191|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200": {
+      "content": "<|reserved_192|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "201": {
+      "content": "<|reserved_193|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "202": {
+      "content": "<|reserved_194|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "203": {
+      "content": "<|reserved_195|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "204": {
+      "content": "<|reserved_196|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "205": {
+      "content": "<|reserved_197|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "206": {
+      "content": "<|reserved_198|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "207": {
+      "content": "<|reserved_199|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "208": {
+      "content": "<|reserved_200|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "209": {
+      "content": "<|reserved_201|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "210": {
+      "content": "<|reserved_202|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "211": {
+      "content": "<|reserved_203|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "212": {
+      "content": "<|reserved_204|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "213": {
+      "content": "<|reserved_205|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "214": {
+      "content": "<|reserved_206|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "215": {
+      "content": "<|reserved_207|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "216": {
+      "content": "<|reserved_208|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "217": {
+      "content": "<|reserved_209|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "218": {
+      "content": "<|reserved_210|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "219": {
+      "content": "<|reserved_211|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "220": {
+      "content": "<|reserved_212|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "221": {
+      "content": "<|reserved_213|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "222": {
+      "content": "<|reserved_214|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "223": {
+      "content": "<|reserved_215|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "224": {
+      "content": "<|reserved_216|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "225": {
+      "content": "<|reserved_217|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "226": {
+      "content": "<|reserved_218|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "227": {
+      "content": "<|reserved_219|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "228": {
+      "content": "<|reserved_220|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "229": {
+      "content": "<|reserved_221|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "230": {
+      "content": "<|reserved_222|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "231": {
+      "content": "<|reserved_223|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "232": {
+      "content": "<|reserved_224|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "233": {
+      "content": "<|reserved_225|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "234": {
+      "content": "<|reserved_226|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "235": {
+      "content": "<|reserved_227|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "236": {
+      "content": "<|reserved_228|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "237": {
+      "content": "<|reserved_229|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "238": {
+      "content": "<|reserved_230|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "239": {
+      "content": "<|reserved_231|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "240": {
+      "content": "<|reserved_232|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "241": {
+      "content": "<|reserved_233|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "242": {
+      "content": "<|reserved_234|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "243": {
+      "content": "<|reserved_235|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "244": {
+      "content": "<|reserved_236|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "245": {
+      "content": "<|reserved_237|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "246": {
+      "content": "<|reserved_238|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "247": {
+      "content": "<|reserved_239|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "248": {
+      "content": "<|reserved_240|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "249": {
+      "content": "<|reserved_241|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250": {
+      "content": "<|reserved_242|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "251": {
+      "content": "<|reserved_243|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "252": {
+      "content": "<|reserved_244|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "253": {
+      "content": "<|reserved_245|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "254": {
+      "content": "<|reserved_246|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "255": {
+      "content": "<|reserved_247|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "256": {
+      "content": "<|reserved_248|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "257": {
+      "content": "<|reserved_249|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "258": {
+      "content": "<|reserved_250|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "259": {
+      "content": "<|reserved_251|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "260": {
+      "content": "<|reserved_252|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "261": {
+      "content": "<|reserved_253|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "262": {
+      "content": "<|reserved_254|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "263": {
+      "content": "<|reserved_255|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|reserved_0|>",
+    "<|reserved_1|>",
+    "<|reserved_2|>",
+    "<|reserved_3|>",
+    "<|reserved_4|>",
+    "<|reserved_5|>",
+    "<|reserved_6|>",
+    "<|reserved_7|>",
+    "<|reserved_8|>",
+    "<|reserved_9|>",
+    "<|reserved_10|>",
+    "<|reserved_11|>",
+    "<|reserved_12|>",
+    "<|reserved_13|>",
+    "<|reserved_14|>",
+    "<|reserved_15|>",
+    "<|reserved_16|>",
+    "<|reserved_17|>",
+    "<|reserved_18|>",
+    "<|reserved_19|>",
+    "<|reserved_20|>",
+    "<|reserved_21|>",
+    "<|reserved_22|>",
+    "<|reserved_23|>",
+    "<|reserved_24|>",
+    "<|reserved_25|>",
+    "<|reserved_26|>",
+    "<|reserved_27|>",
+    "<|reserved_28|>",
+    "<|reserved_29|>",
+    "<|reserved_30|>",
+    "<|reserved_31|>",
+    "<|reserved_32|>",
+    "<|reserved_33|>",
+    "<|reserved_34|>",
+    "<|reserved_35|>",
+    "<|reserved_36|>",
+    "<|reserved_37|>",
+    "<|reserved_38|>",
+    "<|reserved_39|>",
+    "<|reserved_40|>",
+    "<|reserved_41|>",
+    "<|reserved_42|>",
+    "<|reserved_43|>",
+    "<|reserved_44|>",
+    "<|reserved_45|>",
+    "<|reserved_46|>",
+    "<|reserved_47|>",
+    "<|reserved_48|>",
+    "<|reserved_49|>",
+    "<|reserved_50|>",
+    "<|reserved_51|>",
+    "<|reserved_52|>",
+    "<|reserved_53|>",
+    "<|reserved_54|>",
+    "<|reserved_55|>",
+    "<|reserved_56|>",
+    "<|reserved_57|>",
+    "<|reserved_58|>",
+    "<|reserved_59|>",
+    "<|reserved_60|>",
+    "<|reserved_61|>",
+    "<|reserved_62|>",
+    "<|reserved_63|>",
+    "<|reserved_64|>",
+    "<|reserved_65|>",
+    "<|reserved_66|>",
+    "<|reserved_67|>",
+    "<|reserved_68|>",
+    "<|reserved_69|>",
+    "<|reserved_70|>",
+    "<|reserved_71|>",
+    "<|reserved_72|>",
+    "<|reserved_73|>",
+    "<|reserved_74|>",
+    "<|reserved_75|>",
+    "<|reserved_76|>",
+    "<|reserved_77|>",
+    "<|reserved_78|>",
+    "<|reserved_79|>",
+    "<|reserved_80|>",
+    "<|reserved_81|>",
+    "<|reserved_82|>",
+    "<|reserved_83|>",
+    "<|reserved_84|>",
+    "<|reserved_85|>",
+    "<|reserved_86|>",
+    "<|reserved_87|>",
+    "<|reserved_88|>",
+    "<|reserved_89|>",
+    "<|reserved_90|>",
+    "<|reserved_91|>",
+    "<|reserved_92|>",
+    "<|reserved_93|>",
+    "<|reserved_94|>",
+    "<|reserved_95|>",
+    "<|reserved_96|>",
+    "<|reserved_97|>",
+    "<|reserved_98|>",
+    "<|reserved_99|>",
+    "<|reserved_100|>",
+    "<|reserved_101|>",
+    "<|reserved_102|>",
+    "<|reserved_103|>",
+    "<|reserved_104|>",
+    "<|reserved_105|>",
+    "<|reserved_106|>",
+    "<|reserved_107|>",
+    "<|reserved_108|>",
+    "<|reserved_109|>",
+    "<|reserved_110|>",
+    "<|reserved_111|>",
+    "<|reserved_112|>",
+    "<|reserved_113|>",
+    "<|reserved_114|>",
+    "<|reserved_115|>",
+    "<|reserved_116|>",
+    "<|reserved_117|>",
+    "<|reserved_118|>",
+    "<|reserved_119|>",
+    "<|reserved_120|>",
+    "<|reserved_121|>",
+    "<|reserved_122|>",
+    "<|reserved_123|>",
+    "<|reserved_124|>",
+    "<|reserved_125|>",
+    "<|reserved_126|>",
+    "<|reserved_127|>",
+    "<|reserved_128|>",
+    "<|reserved_129|>",
+    "<|reserved_130|>",
+    "<|reserved_131|>",
+    "<|reserved_132|>",
+    "<|reserved_133|>",
+    "<|reserved_134|>",
+    "<|reserved_135|>",
+    "<|reserved_136|>",
+    "<|reserved_137|>",
+    "<|reserved_138|>",
+    "<|reserved_139|>",
+    "<|reserved_140|>",
+    "<|reserved_141|>",
+    "<|reserved_142|>",
+    "<|reserved_143|>",
+    "<|reserved_144|>",
+    "<|reserved_145|>",
+    "<|reserved_146|>",
+    "<|reserved_147|>",
+    "<|reserved_148|>",
+    "<|reserved_149|>",
+    "<|reserved_150|>",
+    "<|reserved_151|>",
+    "<|reserved_152|>",
+    "<|reserved_153|>",
+    "<|reserved_154|>",
+    "<|reserved_155|>",
+    "<|reserved_156|>",
+    "<|reserved_157|>",
+    "<|reserved_158|>",
+    "<|reserved_159|>",
+    "<|reserved_160|>",
+    "<|reserved_161|>",
+    "<|reserved_162|>",
+    "<|reserved_163|>",
+    "<|reserved_164|>",
+    "<|reserved_165|>",
+    "<|reserved_166|>",
+    "<|reserved_167|>",
+    "<|reserved_168|>",
+    "<|reserved_169|>",
+    "<|reserved_170|>",
+    "<|reserved_171|>",
+    "<|reserved_172|>",
+    "<|reserved_173|>",
+    "<|reserved_174|>",
+    "<|reserved_175|>",
+    "<|reserved_176|>",
+    "<|reserved_177|>",
+    "<|reserved_178|>",
+    "<|reserved_179|>",
+    "<|reserved_180|>",
+    "<|reserved_181|>",
+    "<|reserved_182|>",
+    "<|reserved_183|>",
+    "<|reserved_184|>",
+    "<|reserved_185|>",
+    "<|reserved_186|>",
+    "<|reserved_187|>",
+    "<|reserved_188|>",
+    "<|reserved_189|>",
+    "<|reserved_190|>",
+    "<|reserved_191|>",
+    "<|reserved_192|>",
+    "<|reserved_193|>",
+    "<|reserved_194|>",
+    "<|reserved_195|>",
+    "<|reserved_196|>",
+    "<|reserved_197|>",
+    "<|reserved_198|>",
+    "<|reserved_199|>",
+    "<|reserved_200|>",
+    "<|reserved_201|>",
+    "<|reserved_202|>",
+    "<|reserved_203|>",
+    "<|reserved_204|>",
+    "<|reserved_205|>",
+    "<|reserved_206|>",
+    "<|reserved_207|>",
+    "<|reserved_208|>",
+    "<|reserved_209|>",
+    "<|reserved_210|>",
+    "<|reserved_211|>",
+    "<|reserved_212|>",
+    "<|reserved_213|>",
+    "<|reserved_214|>",
+    "<|reserved_215|>",
+    "<|reserved_216|>",
+    "<|reserved_217|>",
+    "<|reserved_218|>",
+    "<|reserved_219|>",
+    "<|reserved_220|>",
+    "<|reserved_221|>",
+    "<|reserved_222|>",
+    "<|reserved_223|>",
+    "<|reserved_224|>",
+    "<|reserved_225|>",
+    "<|reserved_226|>",
+    "<|reserved_227|>",
+    "<|reserved_228|>",
+    "<|reserved_229|>",
+    "<|reserved_230|>",
+    "<|reserved_231|>",
+    "<|reserved_232|>",
+    "<|reserved_233|>",
+    "<|reserved_234|>",
+    "<|reserved_235|>",
+    "<|reserved_236|>",
+    "<|reserved_237|>",
+    "<|reserved_238|>",
+    "<|reserved_239|>",
+    "<|reserved_240|>",
+    "<|reserved_241|>",
+    "<|reserved_242|>",
+    "<|reserved_243|>",
+    "<|reserved_244|>",
+    "<|reserved_245|>",
+    "<|reserved_246|>",
+    "<|reserved_247|>",
+    "<|reserved_248|>",
+    "<|reserved_249|>",
+    "<|reserved_250|>",
+    "<|reserved_251|>",
+    "<|reserved_252|>",
+    "<|reserved_253|>",
+    "<|reserved_254|>",
+    "<|reserved_255|>"
+  ],
+  "bos_token": "<|bos|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|eos|>",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}

tokenizer_contract.json

@@ -0,0 +1,272 @@
+{
+  "name": "VietnameseTokenizer-v1",
+  "vocab_size": 48000,
+  "special_token_ids": {
+    "<|pad|>": 0,
+    "<|bos|>": 1,
+    "<|eos|>": 2,
+    "<|im_start|>": 3,
+    "<|im_end|>": 4,
+    "<|fim_prefix|>": 5,
+    "<|fim_middle|>": 6,
+    "<|fim_suffix|>": 7,
+    "<|reserved_0|>": 8,
+    "<|reserved_1|>": 9,
+    "<|reserved_2|>": 10,
+    "<|reserved_3|>": 11,
+    "<|reserved_4|>": 12,
+    "<|reserved_5|>": 13,
+    "<|reserved_6|>": 14,
+    "<|reserved_7|>": 15,
+    "<|reserved_8|>": 16,
+    "<|reserved_9|>": 17,
+    "<|reserved_10|>": 18,
+    "<|reserved_11|>": 19,
+    "<|reserved_12|>": 20,
+    "<|reserved_13|>": 21,
+    "<|reserved_14|>": 22,
+    "<|reserved_15|>": 23,
+    "<|reserved_16|>": 24,
+    "<|reserved_17|>": 25,
+    "<|reserved_18|>": 26,
+    "<|reserved_19|>": 27,
+    "<|reserved_20|>": 28,
+    "<|reserved_21|>": 29,
+    "<|reserved_22|>": 30,
+    "<|reserved_23|>": 31,
+    "<|reserved_24|>": 32,
+    "<|reserved_25|>": 33,
+    "<|reserved_26|>": 34,
+    "<|reserved_27|>": 35,
+    "<|reserved_28|>": 36,
+    "<|reserved_29|>": 37,
+    "<|reserved_30|>": 38,
+    "<|reserved_31|>": 39,
+    "<|reserved_32|>": 40,
+    "<|reserved_33|>": 41,
+    "<|reserved_34|>": 42,
+    "<|reserved_35|>": 43,
+    "<|reserved_36|>": 44,
+    "<|reserved_37|>": 45,
+    "<|reserved_38|>": 46,
+    "<|reserved_39|>": 47,
+    "<|reserved_40|>": 48,
+    "<|reserved_41|>": 49,
+    "<|reserved_42|>": 50,
+    "<|reserved_43|>": 51,
+    "<|reserved_44|>": 52,
+    "<|reserved_45|>": 53,
+    "<|reserved_46|>": 54,
+    "<|reserved_47|>": 55,
+    "<|reserved_48|>": 56,
+    "<|reserved_49|>": 57,
+    "<|reserved_50|>": 58,
+    "<|reserved_51|>": 59,
+    "<|reserved_52|>": 60,
+    "<|reserved_53|>": 61,
+    "<|reserved_54|>": 62,
+    "<|reserved_55|>": 63,
+    "<|reserved_56|>": 64,
+    "<|reserved_57|>": 65,
+    "<|reserved_58|>": 66,
+    "<|reserved_59|>": 67,
+    "<|reserved_60|>": 68,
+    "<|reserved_61|>": 69,
+    "<|reserved_62|>": 70,
+    "<|reserved_63|>": 71,
+    "<|reserved_64|>": 72,
+    "<|reserved_65|>": 73,
+    "<|reserved_66|>": 74,
+    "<|reserved_67|>": 75,
+    "<|reserved_68|>": 76,
+    "<|reserved_69|>": 77,
+    "<|reserved_70|>": 78,
+    "<|reserved_71|>": 79,
+    "<|reserved_72|>": 80,
+    "<|reserved_73|>": 81,
+    "<|reserved_74|>": 82,
+    "<|reserved_75|>": 83,
+    "<|reserved_76|>": 84,
+    "<|reserved_77|>": 85,
+    "<|reserved_78|>": 86,
+    "<|reserved_79|>": 87,
+    "<|reserved_80|>": 88,
+    "<|reserved_81|>": 89,
+    "<|reserved_82|>": 90,
+    "<|reserved_83|>": 91,
+    "<|reserved_84|>": 92,
+    "<|reserved_85|>": 93,
+    "<|reserved_86|>": 94,
+    "<|reserved_87|>": 95,
+    "<|reserved_88|>": 96,
+    "<|reserved_89|>": 97,
+    "<|reserved_90|>": 98,
+    "<|reserved_91|>": 99,
+    "<|reserved_92|>": 100,
+    "<|reserved_93|>": 101,
+    "<|reserved_94|>": 102,
+    "<|reserved_95|>": 103,
+    "<|reserved_96|>": 104,
+    "<|reserved_97|>": 105,
+    "<|reserved_98|>": 106,
+    "<|reserved_99|>": 107,
+    "<|reserved_100|>": 108,
+    "<|reserved_101|>": 109,
+    "<|reserved_102|>": 110,
+    "<|reserved_103|>": 111,
+    "<|reserved_104|>": 112,
+    "<|reserved_105|>": 113,
+    "<|reserved_106|>": 114,
+    "<|reserved_107|>": 115,
+    "<|reserved_108|>": 116,
+    "<|reserved_109|>": 117,
+    "<|reserved_110|>": 118,
+    "<|reserved_111|>": 119,
+    "<|reserved_112|>": 120,
+    "<|reserved_113|>": 121,
+    "<|reserved_114|>": 122,
+    "<|reserved_115|>": 123,
+    "<|reserved_116|>": 124,
+    "<|reserved_117|>": 125,
+    "<|reserved_118|>": 126,
+    "<|reserved_119|>": 127,
+    "<|reserved_120|>": 128,
+    "<|reserved_121|>": 129,
+    "<|reserved_122|>": 130,
+    "<|reserved_123|>": 131,
+    "<|reserved_124|>": 132,
+    "<|reserved_125|>": 133,
+    "<|reserved_126|>": 134,
+    "<|reserved_127|>": 135,
+    "<|reserved_128|>": 136,
+    "<|reserved_129|>": 137,
+    "<|reserved_130|>": 138,
+    "<|reserved_131|>": 139,
+    "<|reserved_132|>": 140,
+    "<|reserved_133|>": 141,
+    "<|reserved_134|>": 142,
+    "<|reserved_135|>": 143,
+    "<|reserved_136|>": 144,
+    "<|reserved_137|>": 145,
+    "<|reserved_138|>": 146,
+    "<|reserved_139|>": 147,
+    "<|reserved_140|>": 148,
+    "<|reserved_141|>": 149,
+    "<|reserved_142|>": 150,
+    "<|reserved_143|>": 151,
+    "<|reserved_144|>": 152,
+    "<|reserved_145|>": 153,
+    "<|reserved_146|>": 154,
+    "<|reserved_147|>": 155,
+    "<|reserved_148|>": 156,
+    "<|reserved_149|>": 157,
+    "<|reserved_150|>": 158,
+    "<|reserved_151|>": 159,
+    "<|reserved_152|>": 160,
+    "<|reserved_153|>": 161,
+    "<|reserved_154|>": 162,
+    "<|reserved_155|>": 163,
+    "<|reserved_156|>": 164,
+    "<|reserved_157|>": 165,
+    "<|reserved_158|>": 166,
+    "<|reserved_159|>": 167,
+    "<|reserved_160|>": 168,
+    "<|reserved_161|>": 169,
+    "<|reserved_162|>": 170,
+    "<|reserved_163|>": 171,
+    "<|reserved_164|>": 172,
+    "<|reserved_165|>": 173,
+    "<|reserved_166|>": 174,
+    "<|reserved_167|>": 175,
+    "<|reserved_168|>": 176,
+    "<|reserved_169|>": 177,
+    "<|reserved_170|>": 178,
+    "<|reserved_171|>": 179,
+    "<|reserved_172|>": 180,
+    "<|reserved_173|>": 181,
+    "<|reserved_174|>": 182,
+    "<|reserved_175|>": 183,
+    "<|reserved_176|>": 184,
+    "<|reserved_177|>": 185,
+    "<|reserved_178|>": 186,
+    "<|reserved_179|>": 187,
+    "<|reserved_180|>": 188,
+    "<|reserved_181|>": 189,
+    "<|reserved_182|>": 190,
+    "<|reserved_183|>": 191,
+    "<|reserved_184|>": 192,
+    "<|reserved_185|>": 193,
+    "<|reserved_186|>": 194,
+    "<|reserved_187|>": 195,
+    "<|reserved_188|>": 196,
+    "<|reserved_189|>": 197,
+    "<|reserved_190|>": 198,
+    "<|reserved_191|>": 199,
+    "<|reserved_192|>": 200,
+    "<|reserved_193|>": 201,
+    "<|reserved_194|>": 202,
+    "<|reserved_195|>": 203,
+    "<|reserved_196|>": 204,
+    "<|reserved_197|>": 205,
+    "<|reserved_198|>": 206,
+    "<|reserved_199|>": 207,
+    "<|reserved_200|>": 208,
+    "<|reserved_201|>": 209,
+    "<|reserved_202|>": 210,
+    "<|reserved_203|>": 211,
+    "<|reserved_204|>": 212,
+    "<|reserved_205|>": 213,
+    "<|reserved_206|>": 214,
+    "<|reserved_207|>": 215,
+    "<|reserved_208|>": 216,
+    "<|reserved_209|>": 217,
+    "<|reserved_210|>": 218,
+    "<|reserved_211|>": 219,
+    "<|reserved_212|>": 220,
+    "<|reserved_213|>": 221,
+    "<|reserved_214|>": 222,
+    "<|reserved_215|>": 223,
+    "<|reserved_216|>": 224,
+    "<|reserved_217|>": 225,
+    "<|reserved_218|>": 226,
+    "<|reserved_219|>": 227,
+    "<|reserved_220|>": 228,
+    "<|reserved_221|>": 229,
+    "<|reserved_222|>": 230,
+    "<|reserved_223|>": 231,
+    "<|reserved_224|>": 232,
+    "<|reserved_225|>": 233,
+    "<|reserved_226|>": 234,
+    "<|reserved_227|>": 235,
+    "<|reserved_228|>": 236,
+    "<|reserved_229|>": 237,
+    "<|reserved_230|>": 238,
+    "<|reserved_231|>": 239,
+    "<|reserved_232|>": 240,
+    "<|reserved_233|>": 241,
+    "<|reserved_234|>": 242,
+    "<|reserved_235|>": 243,
+    "<|reserved_236|>": 244,
+    "<|reserved_237|>": 245,
+    "<|reserved_238|>": 246,
+    "<|reserved_239|>": 247,
+    "<|reserved_240|>": 248,
+    "<|reserved_241|>": 249,
+    "<|reserved_242|>": 250,
+    "<|reserved_243|>": 251,
+    "<|reserved_244|>": 252,
+    "<|reserved_245|>": 253,
+    "<|reserved_246|>": 254,
+    "<|reserved_247|>": 255,
+    "<|reserved_248|>": 256,
+    "<|reserved_249|>": 257,
+    "<|reserved_250|>": 258,
+    "<|reserved_251|>": 259,
+    "<|reserved_252|>": 260,
+    "<|reserved_253|>": 261,
+    "<|reserved_254|>": 262,
+    "<|reserved_255|>": 263
+  },
+  "normalization": "NFC",
+  "algorithm": "Byte-level BPE"
+}