Text Generation
Transformers
Vietnamese
English
tokenizer
byte-level-bpe
vietnamese
english
code
qwen-style
conversational
Instructions to use PaxiAI/Vietnamese-Tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use PaxiAI/Vietnamese-Tokenizer with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="PaxiAI/Vietnamese-Tokenizer") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("PaxiAI/Vietnamese-Tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use PaxiAI/Vietnamese-Tokenizer with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "PaxiAI/Vietnamese-Tokenizer" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "PaxiAI/Vietnamese-Tokenizer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/PaxiAI/Vietnamese-Tokenizer
- SGLang
How to use PaxiAI/Vietnamese-Tokenizer with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "PaxiAI/Vietnamese-Tokenizer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "PaxiAI/Vietnamese-Tokenizer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "PaxiAI/Vietnamese-Tokenizer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "PaxiAI/Vietnamese-Tokenizer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use PaxiAI/Vietnamese-Tokenizer with Docker Model Runner:
docker model run hf.co/PaxiAI/Vietnamese-Tokenizer
Download tokenizer_config.json from PaxiAI/Vietnamese-Tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 53.7 kB
-
https://huggingface.co/PaxiAI/Vietnamese-Tokenizer/resolve/main/tokenizer_config.json
- Command line
-
hf download hf://PaxiAI/Vietnamese-Tokenizer/tokenizer_config.json
-
curl -L -o tokenizer_config.json https://huggingface.co/PaxiAI/Vietnamese-Tokenizer/resolve/main/tokenizer_config.json
53.7 kB
| { | |
| "added_tokens_decoder": { | |
| "0": { | |
| "content": "<|pad|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "1": { | |
| "content": "<|bos|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "2": { | |
| "content": "<|eos|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "3": { | |
| "content": "<|im_start|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "4": { | |
| "content": "<|im_end|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "5": { | |
| "content": "<|fim_prefix|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "6": { | |
| "content": "<|fim_middle|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "7": { | |
| "content": "<|fim_suffix|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "8": { | |
| "content": "<|reserved_0|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "9": { | |
| "content": "<|reserved_1|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "10": { | |
| "content": "<|reserved_2|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "11": { | |
| "content": "<|reserved_3|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "12": { | |
| "content": "<|reserved_4|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "13": { | |
| "content": "<|reserved_5|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "14": { | |
| "content": "<|reserved_6|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "15": { | |
| "content": "<|reserved_7|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "16": { | |
| "content": "<|reserved_8|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "17": { | |
| "content": "<|reserved_9|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "18": { | |
| "content": "<|reserved_10|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "19": { | |
| "content": "<|reserved_11|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "20": { | |
| "content": "<|reserved_12|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "21": { | |
| "content": "<|reserved_13|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "22": { | |
| "content": "<|reserved_14|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "23": { | |
| "content": "<|reserved_15|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "24": { | |
| "content": "<|reserved_16|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "25": { | |
| "content": "<|reserved_17|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "26": { | |
| "content": "<|reserved_18|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "27": { | |
| "content": "<|reserved_19|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "28": { | |
| "content": "<|reserved_20|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "29": { | |
| "content": "<|reserved_21|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "30": { | |
| "content": "<|reserved_22|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "31": { | |
| "content": "<|reserved_23|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "32": { | |
| "content": "<|reserved_24|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "33": { | |
| "content": "<|reserved_25|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "34": { | |
| "content": "<|reserved_26|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "35": { | |
| "content": "<|reserved_27|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "36": { | |
| "content": "<|reserved_28|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "37": { | |
| "content": "<|reserved_29|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "38": { | |
| "content": "<|reserved_30|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "39": { | |
| "content": "<|reserved_31|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "40": { | |
| "content": "<|reserved_32|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "41": { | |
| "content": "<|reserved_33|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "42": { | |
| "content": "<|reserved_34|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "43": { | |
| "content": "<|reserved_35|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "44": { | |
| "content": "<|reserved_36|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "45": { | |
| "content": "<|reserved_37|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "46": { | |
| "content": "<|reserved_38|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "47": { | |
| "content": "<|reserved_39|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "48": { | |
| "content": "<|reserved_40|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "49": { | |
| "content": "<|reserved_41|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "50": { | |
| "content": "<|reserved_42|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "51": { | |
| "content": "<|reserved_43|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "52": { | |
| "content": "<|reserved_44|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "53": { | |
| "content": "<|reserved_45|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "54": { | |
| "content": "<|reserved_46|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "55": { | |
| "content": "<|reserved_47|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "56": { | |
| "content": "<|reserved_48|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "57": { | |
| "content": "<|reserved_49|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "58": { | |
| "content": "<|reserved_50|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "59": { | |
| "content": "<|reserved_51|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "60": { | |
| "content": "<|reserved_52|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "61": { | |
| "content": "<|reserved_53|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "62": { | |
| "content": "<|reserved_54|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "63": { | |
| "content": "<|reserved_55|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "64": { | |
| "content": "<|reserved_56|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "65": { | |
| "content": "<|reserved_57|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "66": { | |
| "content": "<|reserved_58|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "67": { | |
| "content": "<|reserved_59|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "68": { | |
| "content": "<|reserved_60|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "69": { | |
| "content": "<|reserved_61|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "70": { | |
| "content": "<|reserved_62|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "71": { | |
| "content": "<|reserved_63|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "72": { | |
| "content": "<|reserved_64|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "73": { | |
| "content": "<|reserved_65|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "74": { | |
| "content": "<|reserved_66|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "75": { | |
| "content": "<|reserved_67|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "76": { | |
| "content": "<|reserved_68|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "77": { | |
| "content": "<|reserved_69|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "78": { | |
| "content": "<|reserved_70|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "79": { | |
| "content": "<|reserved_71|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "80": { | |
| "content": "<|reserved_72|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "81": { | |
| "content": "<|reserved_73|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "82": { | |
| "content": "<|reserved_74|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "83": { | |
| "content": "<|reserved_75|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "84": { | |
| "content": "<|reserved_76|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "85": { | |
| "content": "<|reserved_77|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "86": { | |
| "content": "<|reserved_78|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "87": { | |
| "content": "<|reserved_79|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "88": { | |
| "content": "<|reserved_80|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "89": { | |
| "content": "<|reserved_81|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "90": { | |
| "content": "<|reserved_82|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "91": { | |
| "content": "<|reserved_83|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "92": { | |
| "content": "<|reserved_84|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "93": { | |
| "content": "<|reserved_85|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "94": { | |
| "content": "<|reserved_86|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "95": { | |
| "content": "<|reserved_87|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "96": { | |
| "content": "<|reserved_88|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "97": { | |
| "content": "<|reserved_89|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "98": { | |
| "content": "<|reserved_90|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "99": { | |
| "content": "<|reserved_91|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "100": { | |
| "content": "<|reserved_92|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "101": { | |
| "content": "<|reserved_93|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "102": { | |
| "content": "<|reserved_94|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "103": { | |
| "content": "<|reserved_95|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "104": { | |
| "content": "<|reserved_96|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "105": { | |
| "content": "<|reserved_97|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "106": { | |
| "content": "<|reserved_98|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "107": { | |
| "content": "<|reserved_99|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "108": { | |
| "content": "<|reserved_100|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "109": { | |
| "content": "<|reserved_101|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "110": { | |
| "content": "<|reserved_102|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "111": { | |
| "content": "<|reserved_103|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "112": { | |
| "content": "<|reserved_104|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "113": { | |
| "content": "<|reserved_105|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "114": { | |
| "content": "<|reserved_106|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "115": { | |
| "content": "<|reserved_107|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "116": { | |
| "content": "<|reserved_108|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "117": { | |
| "content": "<|reserved_109|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "118": { | |
| "content": "<|reserved_110|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "119": { | |
| "content": "<|reserved_111|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "120": { | |
| "content": "<|reserved_112|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "121": { | |
| "content": "<|reserved_113|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "122": { | |
| "content": "<|reserved_114|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "123": { | |
| "content": "<|reserved_115|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "124": { | |
| "content": "<|reserved_116|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "125": { | |
| "content": "<|reserved_117|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "126": { | |
| "content": "<|reserved_118|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "127": { | |
| "content": "<|reserved_119|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "128": { | |
| "content": "<|reserved_120|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "129": { | |
| "content": "<|reserved_121|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "130": { | |
| "content": "<|reserved_122|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "131": { | |
| "content": "<|reserved_123|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "132": { | |
| "content": "<|reserved_124|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "133": { | |
| "content": "<|reserved_125|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "134": { | |
| "content": "<|reserved_126|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "135": { | |
| "content": "<|reserved_127|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "136": { | |
| "content": "<|reserved_128|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "137": { | |
| "content": "<|reserved_129|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "138": { | |
| "content": "<|reserved_130|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "139": { | |
| "content": "<|reserved_131|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "140": { | |
| "content": "<|reserved_132|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "141": { | |
| "content": "<|reserved_133|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "142": { | |
| "content": "<|reserved_134|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "143": { | |
| "content": "<|reserved_135|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "144": { | |
| "content": "<|reserved_136|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "145": { | |
| "content": "<|reserved_137|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "146": { | |
| "content": "<|reserved_138|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "147": { | |
| "content": "<|reserved_139|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "148": { | |
| "content": "<|reserved_140|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "149": { | |
| "content": "<|reserved_141|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "150": { | |
| "content": "<|reserved_142|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "151": { | |
| "content": "<|reserved_143|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "152": { | |
| "content": "<|reserved_144|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "153": { | |
| "content": "<|reserved_145|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "154": { | |
| "content": "<|reserved_146|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "155": { | |
| "content": "<|reserved_147|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "156": { | |
| "content": "<|reserved_148|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "157": { | |
| "content": "<|reserved_149|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "158": { | |
| "content": "<|reserved_150|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "159": { | |
| "content": "<|reserved_151|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "160": { | |
| "content": "<|reserved_152|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "161": { | |
| "content": "<|reserved_153|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "162": { | |
| "content": "<|reserved_154|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "163": { | |
| "content": "<|reserved_155|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "164": { | |
| "content": "<|reserved_156|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "165": { | |
| "content": "<|reserved_157|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "166": { | |
| "content": "<|reserved_158|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "167": { | |
| "content": "<|reserved_159|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "168": { | |
| "content": "<|reserved_160|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "169": { | |
| "content": "<|reserved_161|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "170": { | |
| "content": "<|reserved_162|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "171": { | |
| "content": "<|reserved_163|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "172": { | |
| "content": "<|reserved_164|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "173": { | |
| "content": "<|reserved_165|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "174": { | |
| "content": "<|reserved_166|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "175": { | |
| "content": "<|reserved_167|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "176": { | |
| "content": "<|reserved_168|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "177": { | |
| "content": "<|reserved_169|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "178": { | |
| "content": "<|reserved_170|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "179": { | |
| "content": "<|reserved_171|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "180": { | |
| "content": "<|reserved_172|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "181": { | |
| "content": "<|reserved_173|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "182": { | |
| "content": "<|reserved_174|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "183": { | |
| "content": "<|reserved_175|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "184": { | |
| "content": "<|reserved_176|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "185": { | |
| "content": "<|reserved_177|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "186": { | |
| "content": "<|reserved_178|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "187": { | |
| "content": "<|reserved_179|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "188": { | |
| "content": "<|reserved_180|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "189": { | |
| "content": "<|reserved_181|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "190": { | |
| "content": "<|reserved_182|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "191": { | |
| "content": "<|reserved_183|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "192": { | |
| "content": "<|reserved_184|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "193": { | |
| "content": "<|reserved_185|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "194": { | |
| "content": "<|reserved_186|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "195": { | |
| "content": "<|reserved_187|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "196": { | |
| "content": "<|reserved_188|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "197": { | |
| "content": "<|reserved_189|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "198": { | |
| "content": "<|reserved_190|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "199": { | |
| "content": "<|reserved_191|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "200": { | |
| "content": "<|reserved_192|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "201": { | |
| "content": "<|reserved_193|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "202": { | |
| "content": "<|reserved_194|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "203": { | |
| "content": "<|reserved_195|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "204": { | |
| "content": "<|reserved_196|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "205": { | |
| "content": "<|reserved_197|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "206": { | |
| "content": "<|reserved_198|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "207": { | |
| "content": "<|reserved_199|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "208": { | |
| "content": "<|reserved_200|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "209": { | |
| "content": "<|reserved_201|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "210": { | |
| "content": "<|reserved_202|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "211": { | |
| "content": "<|reserved_203|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "212": { | |
| "content": "<|reserved_204|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "213": { | |
| "content": "<|reserved_205|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "214": { | |
| "content": "<|reserved_206|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "215": { | |
| "content": "<|reserved_207|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "216": { | |
| "content": "<|reserved_208|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "217": { | |
| "content": "<|reserved_209|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "218": { | |
| "content": "<|reserved_210|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "219": { | |
| "content": "<|reserved_211|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "220": { | |
| "content": "<|reserved_212|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "221": { | |
| "content": "<|reserved_213|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "222": { | |
| "content": "<|reserved_214|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "223": { | |
| "content": "<|reserved_215|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "224": { | |
| "content": "<|reserved_216|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "225": { | |
| "content": "<|reserved_217|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "226": { | |
| "content": "<|reserved_218|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "227": { | |
| "content": "<|reserved_219|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "228": { | |
| "content": "<|reserved_220|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "229": { | |
| "content": "<|reserved_221|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "230": { | |
| "content": "<|reserved_222|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "231": { | |
| "content": "<|reserved_223|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "232": { | |
| "content": "<|reserved_224|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "233": { | |
| "content": "<|reserved_225|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "234": { | |
| "content": "<|reserved_226|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "235": { | |
| "content": "<|reserved_227|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "236": { | |
| "content": "<|reserved_228|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "237": { | |
| "content": "<|reserved_229|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "238": { | |
| "content": "<|reserved_230|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "239": { | |
| "content": "<|reserved_231|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "240": { | |
| "content": "<|reserved_232|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "241": { | |
| "content": "<|reserved_233|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "242": { | |
| "content": "<|reserved_234|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "243": { | |
| "content": "<|reserved_235|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "244": { | |
| "content": "<|reserved_236|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "245": { | |
| "content": "<|reserved_237|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "246": { | |
| "content": "<|reserved_238|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "247": { | |
| "content": "<|reserved_239|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "248": { | |
| "content": "<|reserved_240|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "249": { | |
| "content": "<|reserved_241|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "250": { | |
| "content": "<|reserved_242|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "251": { | |
| "content": "<|reserved_243|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "252": { | |
| "content": "<|reserved_244|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "253": { | |
| "content": "<|reserved_245|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "254": { | |
| "content": "<|reserved_246|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "255": { | |
| "content": "<|reserved_247|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "256": { | |
| "content": "<|reserved_248|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "257": { | |
| "content": "<|reserved_249|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "258": { | |
| "content": "<|reserved_250|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "259": { | |
| "content": "<|reserved_251|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "260": { | |
| "content": "<|reserved_252|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "261": { | |
| "content": "<|reserved_253|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "262": { | |
| "content": "<|reserved_254|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| }, | |
| "263": { | |
| "content": "<|reserved_255|>", | |
| "lstrip": false, | |
| "normalized": false, | |
| "rstrip": false, | |
| "single_word": false, | |
| "special": true | |
| } | |
| }, | |
| "additional_special_tokens": [ | |
| "<|im_start|>", | |
| "<|im_end|>", | |
| "<|fim_prefix|>", | |
| "<|fim_middle|>", | |
| "<|fim_suffix|>", | |
| "<|reserved_0|>", | |
| "<|reserved_1|>", | |
| "<|reserved_2|>", | |
| "<|reserved_3|>", | |
| "<|reserved_4|>", | |
| "<|reserved_5|>", | |
| "<|reserved_6|>", | |
| "<|reserved_7|>", | |
| "<|reserved_8|>", | |
| "<|reserved_9|>", | |
| "<|reserved_10|>", | |
| "<|reserved_11|>", | |
| "<|reserved_12|>", | |
| "<|reserved_13|>", | |
| "<|reserved_14|>", | |
| "<|reserved_15|>", | |
| "<|reserved_16|>", | |
| "<|reserved_17|>", | |
| "<|reserved_18|>", | |
| "<|reserved_19|>", | |
| "<|reserved_20|>", | |
| "<|reserved_21|>", | |
| "<|reserved_22|>", | |
| "<|reserved_23|>", | |
| "<|reserved_24|>", | |
| "<|reserved_25|>", | |
| "<|reserved_26|>", | |
| "<|reserved_27|>", | |
| "<|reserved_28|>", | |
| "<|reserved_29|>", | |
| "<|reserved_30|>", | |
| "<|reserved_31|>", | |
| "<|reserved_32|>", | |
| "<|reserved_33|>", | |
| "<|reserved_34|>", | |
| "<|reserved_35|>", | |
| "<|reserved_36|>", | |
| "<|reserved_37|>", | |
| "<|reserved_38|>", | |
| "<|reserved_39|>", | |
| "<|reserved_40|>", | |
| "<|reserved_41|>", | |
| "<|reserved_42|>", | |
| "<|reserved_43|>", | |
| "<|reserved_44|>", | |
| "<|reserved_45|>", | |
| "<|reserved_46|>", | |
| "<|reserved_47|>", | |
| "<|reserved_48|>", | |
| "<|reserved_49|>", | |
| "<|reserved_50|>", | |
| "<|reserved_51|>", | |
| "<|reserved_52|>", | |
| "<|reserved_53|>", | |
| "<|reserved_54|>", | |
| "<|reserved_55|>", | |
| "<|reserved_56|>", | |
| "<|reserved_57|>", | |
| "<|reserved_58|>", | |
| "<|reserved_59|>", | |
| "<|reserved_60|>", | |
| "<|reserved_61|>", | |
| "<|reserved_62|>", | |
| "<|reserved_63|>", | |
| "<|reserved_64|>", | |
| "<|reserved_65|>", | |
| "<|reserved_66|>", | |
| "<|reserved_67|>", | |
| "<|reserved_68|>", | |
| "<|reserved_69|>", | |
| "<|reserved_70|>", | |
| "<|reserved_71|>", | |
| "<|reserved_72|>", | |
| "<|reserved_73|>", | |
| "<|reserved_74|>", | |
| "<|reserved_75|>", | |
| "<|reserved_76|>", | |
| "<|reserved_77|>", | |
| "<|reserved_78|>", | |
| "<|reserved_79|>", | |
| "<|reserved_80|>", | |
| "<|reserved_81|>", | |
| "<|reserved_82|>", | |
| "<|reserved_83|>", | |
| "<|reserved_84|>", | |
| "<|reserved_85|>", | |
| "<|reserved_86|>", | |
| "<|reserved_87|>", | |
| "<|reserved_88|>", | |
| "<|reserved_89|>", | |
| "<|reserved_90|>", | |
| "<|reserved_91|>", | |
| "<|reserved_92|>", | |
| "<|reserved_93|>", | |
| "<|reserved_94|>", | |
| "<|reserved_95|>", | |
| "<|reserved_96|>", | |
| "<|reserved_97|>", | |
| "<|reserved_98|>", | |
| "<|reserved_99|>", | |
| "<|reserved_100|>", | |
| "<|reserved_101|>", | |
| "<|reserved_102|>", | |
| "<|reserved_103|>", | |
| "<|reserved_104|>", | |
| "<|reserved_105|>", | |
| "<|reserved_106|>", | |
| "<|reserved_107|>", | |
| "<|reserved_108|>", | |
| "<|reserved_109|>", | |
| "<|reserved_110|>", | |
| "<|reserved_111|>", | |
| "<|reserved_112|>", | |
| "<|reserved_113|>", | |
| "<|reserved_114|>", | |
| "<|reserved_115|>", | |
| "<|reserved_116|>", | |
| "<|reserved_117|>", | |
| "<|reserved_118|>", | |
| "<|reserved_119|>", | |
| "<|reserved_120|>", | |
| "<|reserved_121|>", | |
| "<|reserved_122|>", | |
| "<|reserved_123|>", | |
| "<|reserved_124|>", | |
| "<|reserved_125|>", | |
| "<|reserved_126|>", | |
| "<|reserved_127|>", | |
| "<|reserved_128|>", | |
| "<|reserved_129|>", | |
| "<|reserved_130|>", | |
| "<|reserved_131|>", | |
| "<|reserved_132|>", | |
| "<|reserved_133|>", | |
| "<|reserved_134|>", | |
| "<|reserved_135|>", | |
| "<|reserved_136|>", | |
| "<|reserved_137|>", | |
| "<|reserved_138|>", | |
| "<|reserved_139|>", | |
| "<|reserved_140|>", | |
| "<|reserved_141|>", | |
| "<|reserved_142|>", | |
| "<|reserved_143|>", | |
| "<|reserved_144|>", | |
| "<|reserved_145|>", | |
| "<|reserved_146|>", | |
| "<|reserved_147|>", | |
| "<|reserved_148|>", | |
| "<|reserved_149|>", | |
| "<|reserved_150|>", | |
| "<|reserved_151|>", | |
| "<|reserved_152|>", | |
| "<|reserved_153|>", | |
| "<|reserved_154|>", | |
| "<|reserved_155|>", | |
| "<|reserved_156|>", | |
| "<|reserved_157|>", | |
| "<|reserved_158|>", | |
| "<|reserved_159|>", | |
| "<|reserved_160|>", | |
| "<|reserved_161|>", | |
| "<|reserved_162|>", | |
| "<|reserved_163|>", | |
| "<|reserved_164|>", | |
| "<|reserved_165|>", | |
| "<|reserved_166|>", | |
| "<|reserved_167|>", | |
| "<|reserved_168|>", | |
| "<|reserved_169|>", | |
| "<|reserved_170|>", | |
| "<|reserved_171|>", | |
| "<|reserved_172|>", | |
| "<|reserved_173|>", | |
| "<|reserved_174|>", | |
| "<|reserved_175|>", | |
| "<|reserved_176|>", | |
| "<|reserved_177|>", | |
| "<|reserved_178|>", | |
| "<|reserved_179|>", | |
| "<|reserved_180|>", | |
| "<|reserved_181|>", | |
| "<|reserved_182|>", | |
| "<|reserved_183|>", | |
| "<|reserved_184|>", | |
| "<|reserved_185|>", | |
| "<|reserved_186|>", | |
| "<|reserved_187|>", | |
| "<|reserved_188|>", | |
| "<|reserved_189|>", | |
| "<|reserved_190|>", | |
| "<|reserved_191|>", | |
| "<|reserved_192|>", | |
| "<|reserved_193|>", | |
| "<|reserved_194|>", | |
| "<|reserved_195|>", | |
| "<|reserved_196|>", | |
| "<|reserved_197|>", | |
| "<|reserved_198|>", | |
| "<|reserved_199|>", | |
| "<|reserved_200|>", | |
| "<|reserved_201|>", | |
| "<|reserved_202|>", | |
| "<|reserved_203|>", | |
| "<|reserved_204|>", | |
| "<|reserved_205|>", | |
| "<|reserved_206|>", | |
| "<|reserved_207|>", | |
| "<|reserved_208|>", | |
| "<|reserved_209|>", | |
| "<|reserved_210|>", | |
| "<|reserved_211|>", | |
| "<|reserved_212|>", | |
| "<|reserved_213|>", | |
| "<|reserved_214|>", | |
| "<|reserved_215|>", | |
| "<|reserved_216|>", | |
| "<|reserved_217|>", | |
| "<|reserved_218|>", | |
| "<|reserved_219|>", | |
| "<|reserved_220|>", | |
| "<|reserved_221|>", | |
| "<|reserved_222|>", | |
| "<|reserved_223|>", | |
| "<|reserved_224|>", | |
| "<|reserved_225|>", | |
| "<|reserved_226|>", | |
| "<|reserved_227|>", | |
| "<|reserved_228|>", | |
| "<|reserved_229|>", | |
| "<|reserved_230|>", | |
| "<|reserved_231|>", | |
| "<|reserved_232|>", | |
| "<|reserved_233|>", | |
| "<|reserved_234|>", | |
| "<|reserved_235|>", | |
| "<|reserved_236|>", | |
| "<|reserved_237|>", | |
| "<|reserved_238|>", | |
| "<|reserved_239|>", | |
| "<|reserved_240|>", | |
| "<|reserved_241|>", | |
| "<|reserved_242|>", | |
| "<|reserved_243|>", | |
| "<|reserved_244|>", | |
| "<|reserved_245|>", | |
| "<|reserved_246|>", | |
| "<|reserved_247|>", | |
| "<|reserved_248|>", | |
| "<|reserved_249|>", | |
| "<|reserved_250|>", | |
| "<|reserved_251|>", | |
| "<|reserved_252|>", | |
| "<|reserved_253|>", | |
| "<|reserved_254|>", | |
| "<|reserved_255|>" | |
| ], | |
| "bos_token": "<|bos|>", | |
| "clean_up_tokenization_spaces": false, | |
| "eos_token": "<|eos|>", | |
| "extra_special_tokens": {}, | |
| "model_max_length": 131072, | |
| "pad_token": "<|pad|>", | |
| "tokenizer_class": "PreTrainedTokenizerFast" | |
| } | |