Copy files from models/inclusionAI/LLaDA2.2-flash

.gitattributes

@@ -17,7 +17,6 @@
 *.ot filter=lfs diff=lfs merge=lfs -text
 *.parquet filter=lfs diff=lfs merge=lfs -text
 *.pb filter=lfs diff=lfs merge=lfs -text
-*.pdf filter=lfs diff=lfs merge=lfs -text
 *.pickle filter=lfs diff=lfs merge=lfs -text
 *.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
@@ -34,4 +33,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
-tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,32 +1,187 @@
 ---
+license: apache-2.0
 library_name: transformers
-pipeline_tag: text-generation
-base_model: upstage/Solar-Open-100B
 tags:
-- omega-ai
-- text-generation
-- mixture-of-experts
+- dllm
+- diffusion
+- llm
+- text_generation
 ---
 
-# Omega AI 100B
+# LLaDA2.2-flash
 
-Omega AI 100B is an Omega-branded deployment based on Upstage Solar-Open-100B.
+**LLaDA2.2-flash** is an agent-oriented diffusion language model in the LLaDA2 series. By introducing **Levenshtein Editing** (with `DELETE` and `INSERT` control tokens) to diffusion language modeling, it represents the LLaDA2 series' first step in agentic applications, including long-context tool use, multi-turn interaction, and robust error correction.For more information, please refer to our [technical report](https://github.com/inclusionAI/LLaDA2.X/blob/main/LLaDA2_2_tech_report.pdf).
 
-## Base model
-upstage/Solar-Open-100B
 
-## Parameter class
-Approximately 102.6B total parameters, 12B active parameters per token.
+<div align="center">
+  <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*9BoDT6rb1BwAAAAAUbAAAAgAemJ7AQ/original" width="800" />
+</div>
 
-## Modifications
-- Omega AI branding
-- Omega repository identity
-- Original model weights preserved
-- No retraining unless separately documented
+<div align="center">
+  <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*W0wnS7xvKm4AAAAAY-AAAAgAemJ7AQ/original" width="800" />
+</div>
 
-## Provenance
-The underlying model architecture and pretrained weights originate from Upstage Solar-Open-100B.
+---
+
+## 📊 Benchmarks
+The following tables compare **LLaDA2.2-flash** and **Ling-2.6-flash** in terms of agentic benchmark scores and throughput (TPS).
+
+**Agentic benchmark scores**
+
+| Benchmark | LLaDA2.2-flash | Ling-2.6-flash |
+| --- | ---: | ---: |
+| SWE-bench Verified | 49.28 | 61.20<sup>†</sup> |
+| SWE-bench Pro | 30.10 | 31.88 |
+| SWE-bench Multilingual | 25.00 | 33.73 |
+| τ²-Bench | 80.33 | 76.36<sup>†</sup>  |
+| Claw-Eval | 64.22 | 64.56<sup>†</sup>  |
+| PinchBench | 81.66 | 81.30<sup>†</sup>  |
+| MCP-Atlas | 46.21 | 41.12 |
+| BFCL-V4 | 60.78 | 66.81 |
+
+> **LLaDA2.2-flash evaluation setup:** The SWE-bench series was evaluated using the Claude Code scaffold. Across all benchmarks, we used a 128K context window with `temperature=1.0`, `block_length=32`, `threshold=0.5`, and `editing_threshold=0.0`. Each score represents the average of five runs.
+
+<sup>†</sup> The Ling-2.6-flash score on SWE-bench Verified is taken from the Ling and Ring 2.6 Technical Report, where it was obtained using the OpenHands scaffold. The Ling-2.6-flash scores on τ²-Bench, Claw-Eval, and PinchBench are also sourced from the technical report, whereas its SWE-bench Pro and SWE-bench Multilingual scores were evaluated by us using the same Claude Code scaffold as LLaDA2.2-flash.
+
+
+**Throughput (TPS)**
+
+| Benchmark | LLaDA2.2-flash (TPS) | Ling-2.6-flash (TPS) |
+| --- | ---: | ---: |
+| SWE-bench Verified | 519.0 | 303.2 |
+| SWE-bench Pro | 485.3 | 283.4 |
+| SWE-bench Multilingual | 459.5 | 200.6 |
+| τ²-Bench | 592.8 | 334.9 |
+| BFCL-V4 | 703.82 | 331.5 |
+
+> **Ling-2.6-flash evaluation setup:** MTP was enabled with 4 draft tokens.
+
+More results will be released in the upcoming technical report.
+
+---
+
+## 🚀 Highlights
+
++ **Efficient 128K Diffusion Infrastructure**: LLaDA2.2-flash extends the context window to **128K** and introduces **Block Routing**, which bounds MoE expert activation at the diffusion-block level to enable efficient long-context agentic workloads.
+
++ **Levenshtein Editing**: We introduces **DELETE** and **INSERT** control tokens, allowing diffusion decoding to edit sequence structure, remove redundant content, and create insertion slots during parallel generation.
+
++ **Agentic Reinforcement Learning**: We propose **Levenshtein Editing ELBO-based Block-level Policy Optimization (L-EBPO)**, which leverages agentic environmental rewards to train levenshtein editing and error correction in multi-turn tool-use scenarios.
+
+---
+
+## 📦 Model Variants
+
+| Model ID | Description | Hugging Face Link |
+| --- | --- | --- |
+| `inclusionAI/LLaDA2.2-flash` | Agent-oriented MoE diffusion language model with Levenshtein Editing. | [🤗 Model Card](https://huggingface.co/inclusionAI/LLaDA2.2-flash) |
+
+<!-- TODO: Add other LLaDA2.2 variants if available. -->
+
+---
+
+## 🔍 Model Overview
+
+**LLaDA2.2-flash** has the following specifications:
+
++ **Type**: Mixture-of-Experts (MoE) Diffusion Language Model with Levenshtein Editing
++ **Context Length**: 128K tokens
++ **Levenshtein Editing Control Tokens**: `DELETE`, `INSERT`
++ **Total Parameters (Non-Embedding)**: 100B
++ **Number of Layers**: 32
++ **Attention Heads**: 32
++ **Positional Encoding**: Rotary Position Embedding (RoPE)
++ **Vocabulary Size**: 157,184
+
+---
+
+## 🤗 Hugging Face Transformers
+
+Make sure you have `transformers` and its dependencies installed.
+
+<!-- TODO: Verify the final inference API, model path, and recommended generation parameters. -->
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_path = "inclusionAI/LLaDA2.2-flash"
+device = "auto"
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_path,
+    trust_remote_code=True,
+    device_map=device,
+)
+model = model.to(torch.bfloat16)
+model.eval()
+
+tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+
+prompt = """Calculate 1+5-28*0.5-200=?"""
+input_ids = tokenizer.apply_chat_template(
+    [{"role": "user", "content": prompt}],
+    add_generation_prompt=True,
+    tokenize=True,
+    return_tensors="pt",
+).input_ids
+
+generated_tokens = model.generate(
+    inputs=input_ids,
+    eos_early_stop=True,
+    gen_length=512,
+    block_length=32,
+    threshold=0.5,
+    editing_threshold=0.0,
+    temperature=0.0,
+)
+
+generated_answer = tokenizer.decode(
+    generated_tokens[0],
+    skip_special_tokens=True,
+)
+print(generated_answer)
+```
+
+### Best Practices
+
+<!-- TODO: Confirm final recommended values for Speed Mode and Quality Mode. -->
+
+To achieve optimal performance, we recommend starting with the following settings:
+
+1. **Sampling Parameters**: Use `block_length=32`, `temperature=0.0`, `top_p=None`, and `top_k=None` as stable default settings.
+
+2. **Denoising Thresholds**: Tune `threshold`, `editing_threshold`, and `max_post_steps` according to the speed-quality trade-off required by the application. Lower thresholds may improve inference speed but can lead to increased repetition or unstable outputs.
+
+3. **Long-Context Agentic Workloads**: For long-context tool-use and multi-turn agent applications, we recommend using **SGLang** as the serving backend. Please ensure that the serving stack is configured for the 128K context window and the model's MoE diffusion inference requirements.
+
+---
+
+## 🤖 ModelScope
+
+If you are in mainland China, we strongly recommend accessing our model from 🤖 [ModelScope](https://modelscope.cn/models/inclusionAI/LLaDA2.2-flash)
+
+---
+
+## Deployment
+
+### SGLang
+
+SGLang deployment support is coming soon.
+
+---
+
+## 🌐 License
+
+This project is licensed under the terms of the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
+
+---
+
+## 🤝 Contact & Collaboration
+
+For questions, collaboration opportunities, or feedback, please reach out via [Hugging Face](https://huggingface.co/inclusionAI/LLaDA2.2-flash) or open an issue in the [repository](https://github.com/inclusionAI).
+
+Join us in advancing open, efficient, and intelligent diffusion language models for agentic applications.
+
+---
 
-## License
-The model weights are subject to the Upstage Solar License.
-Original license terms and attribution requirements continue to apply.

chat_template.jinja

@@ -1,156 +1,159 @@
-{#- ======== Template Parameters ========  #}
-{%- set add_generation_prompt = add_generation_prompt if add_generation_prompt is defined else true %}
-{%- set default_system_prompt = default_system_prompt if default_system_prompt is defined else true %}
-{%- set reasoning_effort = reasoning_effort if reasoning_effort is defined else "high" %}
-{%- set think_render_option = think_render_option if think_render_option is defined else "lastthink" %}
+{#-
+  LLaDA22 chat template — NO-THINK variant.
+  Usage:
+    start_header: "<role>assistant</role>"
+    end_header: "<|role_end|>"
+    end_learnable_inclusive: true
+#}
 
-{#- ======== System Block State ========  #}
-{%- set sys_ns = namespace(is_first_block=true) -%}
-
-{#- ======== Find last user message index ========  #}
-{%- set last_user_idx = namespace(value=-1) -%}
-{%- for message in messages -%}
-    {%- if message.role == 'user' -%}
-        {%- set last_user_idx.value = loop.index0 -%}
+{#- Message Content Rendering ============================================== #}
+{%- macro render_content(content) -%}
+    {%- if content is string -%}
+        {{- content -}}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping -%}
+                {%- if 'text' in item -%}
+                    {{- item.text -}}
+                {%- elif 'image' in item or 'image_url' in item or item.get('type') == 'image' -%}
+                    {{- raise_exception('Constraint Violation: Image data detected.') -}}
+                {%- elif 'video' in item or item.get('type') == 'video' -%}
+                    {{- raise_exception('Constraint Violation: Video data detected.') -}}
+                {%- else -%}
+                    {{- raise_exception('Invalid mapping structure: Missing "text" key.') -}}
+                {%- endif -%}
+            {%- elif item is string -%}
+                {{- item -}}
+            {%- else -%}
+                {{- raise_exception('Invalid item type: Must be string or mapping.') -}}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- elif content is none or content is undefined -%}
+        {{- '' -}}
+    {%- else -%}
+        {{- raise_exception('Fatal error: Content must be a string, iterable, or None.') -}}
     {%- endif -%}
-{%- endfor -%}
+{%- endmacro -%}
 
-{#- ======== System messages renderers ========  #}
-{%- macro render_system_message(user_system_messages) %}
-    {%- if default_system_prompt %}
-        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
-        {%- set sys_ns.is_first_block = false %}
-        {{- "## Provider System Prompt\n\nYou are Solar Open 100B, a large language model trained by Upstage AI, a Korean startup. Your knowledge cutoff is 2025-07. The current date is " + strftime_now("%Y-%m-%d") + "." }}
-    {%- endif -%}
-    {%- if user_system_messages %}
-        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
-        {%- set sys_ns.is_first_block = false %}
-        {{- "## System Prompt" }}
-        {%- for system_message in user_system_messages %}
-            {{- "\n\n" }}
-            {{- system_message }}
-        {%- endfor %}
-    {%- endif -%}
-{%- endmacro %}
-
-{%- macro render_tool_instruction(tools) %}
-    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
-    {%- set sys_ns.is_first_block = false %}
-    {{- "## Tools\n\n### Tool Call Instruction" }}
-    {{- "\nYou may invoke one or more tools to assist with the user's query. Available tools are provided in JSON Schema format: <|tools:begin|><|tool:begin|><tools-json-object><|tool:end|>...<|tools:end|>\n" }}
-    {{- "\n### Available Tools\n" }}
-    {{- "<|tools:begin|>" }}
-    {%- for tool in tools %}
-        {{- "<|tool:begin|>" }}
-        {{- tool.function | tojson }}
-        {{- "<|tool:end|>" }}
-    {%- endfor %}
-    {{- "<|tools:end|>\n" }}
-    {{- "\n### Tool Call Format\n" }}
-    {{- "For each tool call, return a JSON object with the following structure, enclosed within <|tool_call:begin|> and <|tool_call:end|> tags: \n<|tool_call:begin|><tool-call-id><|tool_call:name|><tool-name><|tool_call:args|><args-json-object><|tool_call:end|>\n" }}
-    {{- "- The <tool-call-id> must be a randomly generated string consisting of 10 lowercase letters (a-z) and/or digits (0-9) (e.g., a1b2c3d4e5)\n" }}
-    {{- "\n### Tool Response Format\n" }}
-    {{- "Each tool is responded by `tool` with the following structure:\n<|tool_response:id|><tool-call-id><|tool_response:name|><tool-name><|tool_response:result|><results><|tool_response:end|>\n" }}
-    {{- "- Ensure the <tool-call-id> matches the corresponding tool call" -}}
-{%- endmacro %}
-
-{%- macro render_json_response_format_instruction(response_format) %}
-    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
-    {%- set sys_ns.is_first_block = false %}
-    {{- "## Output Format Constraint" }}
-    {{- "\n\nYour final response should follow the JSON schema: \n[Start of schema]" }}
-    {{- response_format }}
-    {{- "\n[End of schema]\nPlease ensure your answers adhere to this format and do not contain any unnecessary text." }}
-{%- endmacro %}
-
-{%- macro get_tool_name(messages, tool_call_id) %}
-    {%- for msg in messages -%}
-        {%- if msg.role == 'assistant' and msg.tool_calls -%}
-            {%- for tool_call in msg.tool_calls -%}
-                {%- if tool_call.id == tool_call_id -%}
-                    {{- tool_call.function.name }}
-                {%- endif -%}
-            {%- endfor -%}
+{%- macro render_toolcalls(message, ns_tool) -%}
+    {{- '<|tool_calls_section_begin|>' -}}
+    {%- for tool_call in message.get('tool_calls', []) -%}
+        {%- set original_id = tool_call.get('id', '') -%}
+        {%- set func_name = tool_call.get('function', {}).get('name', 'unknown') -%}
+
+        {%- set tool_id = 'functions.' + func_name + ':' + ns_tool.index|string -%}
+
+        {%- set ns_tool.id_map = ns_tool.id_map + [[original_id, tool_id]] -%}
+        {%- set ns_tool.index = ns_tool.index + 1 -%}
+
+        {%- set args = tool_call.get('function', {}).get('arguments', '') -%}
+        {{- '<|tool_call_begin|>' + tool_id + '<|tool_call_argument_begin|>' -}}
+        {%- if args is string -%}
+            {{- args -}}
+        {%- else -%}
+            {{- args | tojson -}}
         {%- endif -%}
+        {{- '<|tool_call_end|>' -}}
     {%- endfor -%}
-{%- endmacro %}
+    {{- '<|tool_calls_section_end|>' -}}
+{%- endmacro -%}
 
-{%- macro render_tool_arguments(tool_arguments) %}
-    {%- if tool_arguments is mapping -%}
-        {{- tool_arguments | tojson }}
-    {%- else -%}
-        {{- tool_arguments }}
+{%- if not messages -%}
+    {{- raise_exception('No messages provided.') -}}
+{%- endif -%}
+
+{# 1. System Instruction, Model Identity & Tools Injection #}
+{%- set ns_sys = namespace(has_system=false, content='') -%}
+
+
+{%- if messages[0].role == 'system' -%}
+    {%- set ns_sys.has_system = true -%}
+    {%- if ns_sys.content -%}
+        {%- set ns_sys.content = ns_sys.content + '\n\n' -%}
     {%- endif -%}
-{%- endmacro %}
+    {%- set ns_sys.content = ns_sys.content + render_content(messages[0].content) -%}
+{%- endif -%}
 
-{#- ======== Render system message ========  #}
-{%- set ns = namespace(system_messages=[]) -%}
-{%- for message in messages -%}
-    {%- if message.role == 'system' -%}
-        {%- set ns.system_messages = ns.system_messages + [message.content] -%}
+{%- if tools or ns_sys.content -%}
+    {{- '<role>system</role>' -}}
+
+    {%- if ns_sys.content -%}
+        {{- ns_sys.content -}}
+        {%- if tools -%}{{ '\n\n' }}{%- endif -%}
     {%- endif -%}
-{%- endfor -%}
 
-{%- if ns.system_messages or default_system_prompt or tools or response_format -%}
-    {{- "<|begin|>system<|content|>" }}
-        {{- render_system_message(ns.system_messages) }}
-        {%- if tools -%}
-            {{- render_tool_instruction(tools) }}
-        {%- endif %}
-        {%- if response_format -%}
-            {{- render_json_response_format_instruction(response_format) }}
-        {%- endif %}
-    {{- "<|end|>" }}
+    {#- Tool Definition Rendering -#}
+    {%- if tools and tools is iterable and tools is not mapping -%}
+        {{- "<tools>\n" -}}
+        {%- if tools_ts_str -%}
+            {{- tools_ts_str -}}
+        {%- else -%}
+            {%- for tool in tools -%}
+                {{- tool | tojson(ensure_ascii=False) -}}{{ '\n' -}}
+            {%- endfor -%}
+        {%- endif -%}
+        {{- "</tools>\n" -}}
+    {%- endif -%}
+
+    {{- '<|role_end|>' -}}
 {%- endif -%}
 
-{#- ======== Render main messages ========  #}
+{# 2. Main Message Rendering Loop #}
+{%- set ns_tool = namespace(index=0, id_map=[]) -%}
+
 {%- for message in messages -%}
-    {%- if message.role == 'user' -%}
-         {{- "<|begin|>user<|content|>" + message.content + "<|end|>" }}
-    {%- elif message.role == 'tool' -%}
-        {%- set prev_is_tool = loop.index0 > 0 and messages[loop.index0 - 1].role == 'tool' -%}
-        {%- set next_is_tool = loop.index0 < (messages | length - 1) and messages[loop.index0 + 1].role == 'tool' -%}
-        {%- if not prev_is_tool -%}
-            {{- "<|begin|>tool<|tool_response|>" }}
-        {%- endif -%}
-        {{- "<|tool_response:begin|>" + message.tool_call_id + "<|tool_response:name|>" }}
-        {{- get_tool_name(messages, message.tool_call_id) }}
-        {{- "<|tool_response:result|>" }}
-        {{- message.content }}
-        {{- "<|tool_response:end|>" }}
-        {%- if not next_is_tool -%}
-            {{- "<|end|>" }}
-        {%- endif -%}
-    {%- elif message.role == 'assistant' -%}
-        {#- ======== Assistant Thinking ========  #}
-        {%- if think_render_option == "all" -%}
-            {%- if message.reasoning -%}
-                {{- "<|begin|>assistant<|think|>" + message.reasoning + "<|end|>" }}
+    {# Skip the first System message as it is pre-rendered above #}
+    {%- if not (loop.first and message.role == 'system') -%}
+
+        {%- set content_text = render_content(message.content) -%}
+        {%- set role_name = message.get('name') or message.role -%}
+
+        {%- if message.role == 'user' or message.role == 'system' -%}
+            {{- '<role>' + role_name + '</role>' -}}
+            {{- content_text + '<|role_end|>' -}}
+
+        {%- elif message.role == 'assistant' -%}
+            {{- '<role>' + role_name + '</role>' -}}
+
+            {#- NO-THINK: completely ignore reasoning_content, strip <think> from content -#}
+            {%- set final_content = content_text -%}
+
+            {#- Strip embedded <think>...</think> from content if present -#}
+            {%- if '<think>' in final_content and '</think>' in final_content -%}
+                {%- set parts = final_content.split('</think>', 1) -%}
+                {%- set final_content = parts[1].strip() -%}
             {%- endif -%}
-        {%- elif think_render_option == "lastthink" -%}
-            {%- if message.reasoning and loop.index0 > last_user_idx.value -%}
-                {{- "<|begin|>assistant<|think|>" + message.reasoning + "<|end|>" }}
+
+            {#- Output content directly, no think tags -#}
+            {%- if final_content -%}
+                {{- final_content -}}
             {%- endif -%}
-        {%- endif -%}
 
-        {#- ======== Assistant Messages ========  #}
-        {%- if message.tool_calls -%}
-            {{- "<|begin|>assistant<|tool_calls|>" }}
-            {%- for tool_call in message.tool_calls -%}
-                {{- "<|tool_call:begin|>" + tool_call.id +"<|tool_call:name|>" + tool_call.function.name + "<|tool_call:args|>" }}
-                {{- render_tool_arguments(tool_call.function.arguments) }}
-                {{- "<|tool_call:end|>" }}
+            {%- if message.tool_calls -%}
+                {{- render_toolcalls(message, ns_tool) -}}
+            {%- endif -%}
+
+            {{- '<|role_end|>' -}}
+
+        {%- elif message.role == 'tool' -%}
+            {{- '<role>tool</role>' -}}
+            {%- set original_id = message.get('tool_call_id', '') -%}
+
+            {%- set ns_lookup = namespace(mapped_id=original_id) -%}
+            {%- for pair in ns_tool.id_map -%}
+                {%- if pair[0] == original_id -%}
+                    {%- set ns_lookup.mapped_id = pair[1] -%}
+                {%- endif -%}
             {%- endfor -%}
-            {{- "<|calls|>" }}
-        {%- else -%}
-            {{- "<|begin|>assistant<|content|>" + message.content + "<|end|>" }}
+
+            {{- '## Return of ' + ns_lookup.mapped_id + '\n' -}}
+            {{- content_text + '\n<|role_end|>' -}}
         {%- endif -%}
+
     {%- endif -%}
 {%- endfor -%}
 
+{# 3. Generation Prompt Suffix — no think tag #}
 {%- if add_generation_prompt -%}
-    {%- if reasoning_effort in ["low", "minimal"] -%}
-        {{- "<|begin|>assistant<|think|><|end|>" }}
-    {%- endif -%}
-    {{- "<|begin|>assistant" }}
+    {{- '<role>assistant</role>' -}}
 {%- endif -%}

config.json

@@ -1,36 +1,58 @@
 {
-    "model_type": "solar_open",
     "architectures": [
-        "SolarOpenForCausalLM"
+        "LLaDA2MoeModelLM"
     ],
-    "pad_token_id": 2,
-    "bos_token_id": 1,
-    "eos_token_id": 2,
-    "partial_rotary_factor": 1.0,
+    "attention_dropout": 0.0,
+    "auto_map": {
+        "AutoConfig": "configuration_llada2_moe.LLaDA2MoeConfig",
+        "AutoModel": "modeling_llada2_moe.LLaDA2MoeModel",
+        "AutoModelForCausalLM": "modeling_llada2_moe.LLaDA2MoeModelLM"
+    },
+    "num_hidden_layers": 32,
     "hidden_size": 4096,
-    "num_hidden_layers": 48,
-    "num_attention_heads": 64,
-    "head_dim": 128,
-    "num_key_value_heads": 8,
-    "vocab_size": 196608,
-    "intermediate_size": 10240,
-    "moe_intermediate_size": 1280,
-    "rms_norm_eps": 1e-05,
-    "rope_theta": 1000000,
+    "intermediate_size": 9216,
+    "first_k_dense_replace": 1,
+    "hidden_act": "silu",
     "max_position_embeddings": 131072,
-    "n_routed_experts": 128,
-    "n_shared_experts": 1,
+    "model_type": "llada2_moe",
+    "moe_intermediate_size": 1024,
     "norm_topk_prob": true,
-    "routed_scaling_factor": 1.0,
     "num_experts_per_tok": 8,
+    "expert_capacity": 48,
+    "block_size": 32,
+    "norm_head": false,
+    "num_attention_heads": 32,
+    "num_experts": 256,
+    "num_key_value_heads": 4,
+    "rope_theta": 3000000,
+    "rope_scaling": null,
     "tie_word_embeddings": false,
     "torch_dtype": "bfloat16",
-    "first_k_dense_replace": 0,
-    "use_cache": true,
-    "transformers_version": "4.57.1",
-    "rope_scaling": {
-        "type": "yarn",
-        "factor": 2.0,
-        "original_max_position_embeddings": 65536
-    }
+    "transformers_version": "5.2.0",
+    "use_bias": false,
+    "use_rmsnorm": true,
+    "rms_norm_eps": 1e-06,
+    "head_dim": 128,
+    "num_shared_experts": 1,
+    "use_cache": false,
+    "use_qk_norm": true,
+    "use_qkv_bias": false,
+    "embedding_dropout": 0.0,
+    "norm_softmax": false,
+    "output_dropout": 0.0,
+    "vocab_size": 157184,
+    "rotary_dim": 64,
+    "using_split_qkv_in_self_attention": false,
+    "router_dtype": "fp32",
+    "moe_router_enable_expert_bias": true,
+    "routed_scaling_factor": 2.5,
+    "n_group": 8,
+    "topk_group": 4,
+    "score_function": "sigmoid",
+    "initializer_range": 0.02,
+    "max_window_layers": 28,
+    "output_router_logits": false,
+    "pad_token_id": 156892,
+    "partial_rotary_factor": 0.5,
+    "use_sliding_window": false
 }

configuration_llada2_moe.py

@@ -0,0 +1,94 @@
+"""LLaDA2 MoE model configuration"""
+
+from transformers.configuration_utils import PretrainedConfig
+
+
+class LLaDA2MoeConfig(PretrainedConfig):
+    model_type = "llada2_moe"
+
+    def __init__(
+        self,
+        vocab_size=30592,
+        hidden_size=1024,
+        intermediate_size=None,
+        num_hidden_layers=24,
+        num_attention_heads=16,
+        num_key_value_heads=0,
+        hidden_act="silu",
+        use_qkv_bias=False,  # llada2 only
+        use_qk_norm=False,
+        use_bias=True,  # llada2 only
+        rms_norm_eps=1e-05,
+        norm_head=False,  # llada2 only
+        tie_word_embeddings=False,  # PretrainedConfig key, here change default value.
+        embedding_dropout=0.1,
+        attention_dropout=0.1,
+        output_dropout=0.1,
+        initializer_range=0.02,
+        max_position_embeddings=16384,
+        rope_theta=10000.0,
+        use_cache=True,
+        use_sliding_window=False,
+        sliding_window=4096,
+        max_window_layers=28,
+        rope_scaling=None,
+        pad_token_id=126081,
+        num_experts=16,
+        num_shared_experts=0,
+        num_experts_per_tok=2,
+        n_group=8,
+        topk_group=4,
+        routed_scaling_factor=2.5,
+        moe_intermediate_size=None,
+        first_k_dense_replace=0,
+        head_dim=None,
+        output_router_logits=False,
+        partial_rotary_factor=0.5,
+        expert_capacity=48,
+        block_size=32,
+        **kwargs,
+    ):
+        self.num_hidden_layers = num_hidden_layers
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.use_qkv_bias = use_qkv_bias
+        self.use_qk_norm = use_qk_norm
+        self.use_bias = use_bias
+        self.norm_head = norm_head
+        self.rms_norm_eps = rms_norm_eps
+        self.embedding_dropout = embedding_dropout
+        self.attention_dropout = attention_dropout
+        self.output_dropout = output_dropout
+        self.initializer_range = initializer_range
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.use_cache = use_cache
+        self.use_sliding_window = use_sliding_window
+        self.sliding_window = sliding_window
+        self.max_window_layers = max_window_layers
+        self.head_dim = head_dim or self.hidden_size // self.num_attention_heads
+        self.rope_scaling = rope_scaling
+
+        # MoE configs
+        self.num_experts = num_experts
+        self.num_shared_experts = num_shared_experts
+        self.num_experts_per_tok = num_experts_per_tok
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.moe_intermediate_size = moe_intermediate_size
+        self.first_k_dense_replace = first_k_dense_replace
+        self.output_router_logits = output_router_logits
+        self.routed_scaling_factor = routed_scaling_factor
+        self.partial_rotary_factor = partial_rotary_factor
+
+        # Block routing configs
+        self.expert_capacity = expert_capacity
+        self.block_size = block_size
+
+        super().__init__(
+            pad_token_id=pad_token_id, tie_word_embeddings=tie_word_embeddings, **kwargs
+        )

generation_config.json

@@ -1,14 +1,7 @@
 {
   "_from_model_config": true,
-  "bos_token_id": 1,
-  "eos_token_id": [
-    2,
-    24,
-    25
-  ],
-  "pad_token_id": 2,
-  "transformers_version": "4.57.3",
-  "do_sample": true,
-  "temperature": 0.8,
-  "top_p": 0.95
+  "eos_token_id": [156892, 156900],
+  "pad_token_id": 156892,
+  "transformers_version": "5.2.0",
+  "use_cache": false
 }

model-00000-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d1990021fcc7bdbd07c1c0089073e2e6c412e1fd8cc4e40d349846087546127
+size 12304705856

model-00001-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4abb1bc2dff14189ad2ed4b9800af3fb54757a0759ce68aed6204b11ee4a7deb
+size 6241216968

model-00002-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5a69255edc9ec0541f84448cb260da5b5067b45709554ea3e2a450b52eb267a
+size 6241217160

model-00003-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d42ed754dbe4f71a8be830e36e60f896dcf3b843515cd02f44ba99c48b798b94
+size 6241217160

model-00004-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc808529f723a2094b0ed4feb6f73eaa57b41f6e3fc808e953890b9b1581b1ad
+size 6241217160

model-00005-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9b716b15ee2052eb27f367add5c058c2cdc379b4c0d694442320afa1f6d8014
+size 6241217160

model-00006-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ef0ab4f979996160dcedb3ea801ddaff6c76a8bd90743331eda8a38c7a37d66
+size 6241217160

model-00007-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ef1088e751d6f96b611596f82c2a75d40f2dc2ebcc89710cba0694aa44e11b4
+size 6241217160

model-00008-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3b29011b004955d811d126e6079b0a020fad605dfb838d994746fc2d3697386
+size 6241217160

model-00009-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ec94d969a5d31c9263d2f6ea883dee0e23d824bad7f5df6ce3c066541628051
+size 6241217160

model-00010-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edeb684cb0a29468d3292dd79bb349b8eb8b6b2b455da89b31272cfb02abe43d
+size 6241217160

model-00011-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:057bbb40f925141c459c22b34fb29aa43293f9686b99c52eff40a5790d3b97a4
+size 6241217160

model-00012-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5297203652203933efb0e5c1b4231592cd387d3e6a8ba005d6ad718ee1ba5cd5
+size 6241217528

model-00013-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7b52c6ceb64ce9cc192e7153c1003f58feb84a6b4eff66dd8bcf14cbeaca08c3
+size 6241217904

model-00014-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:790c089f45a061711ff6e2442359702b3011e4cf1055928e53175ff30db7da6c
+size 6241217904

model-00015-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1688f8d99143a104fc743959d293c9c7e8ed88ca9246a32b6c66e568250e0e76
+size 6241217904

model-00016-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b673b90d7ce20df76bcd63ed68da1fc56a23c5b6ea66d80003a6b484a8583cb
+size 6241217904

model-00017-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9741de327958cd1542206f46df66c064ae8b55eaf6408c57ec32099097dd22e
+size 6241217904

model-00018-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2b1eedc53f6f5110c9f7cbda3e39682600572aa142b1c8792a50300572336ae
+size 6241217904

model-00019-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ce6cc65fadbe15d0d9695e1a5e371f2df4ae2c6bcb5fcb93ed731a3f4322fc2
+size 6241217904

model-00020-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a173a41b95409b6d1aff76f2c923f51936f9679c07b4480e3220ed5e8aaf6688
+size 6241217904

model-00021-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f369627b48e8ff61d77daca5dddc4fe8e66428156d25030d3c45c522fa212f48
+size 6241217904

model-00022-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1670d3b111e173e45f773e2aef2a982f7e8b074c5a74ce2b9d8a2d795d95a30
+size 6241217904

model-00023-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cb566ee0a2bb9161946dd7dd2785f3ff35f31cd59d26b8e52cd8888bc33126e
+size 6241217904

model-00024-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:110d4e1c4071796e4e6d3562a6afa55a5e9653f79b22d507c9472baea9ab8a0e
+size 6241217904

model-00025-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e50d758bed684abfd4dfe567d41d652908bb809ad4581925d750d4f82b384e12
+size 6241217904

model-00026-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12c25ff4880833827155b08ead89b59ee66cbf1942c7f30a511d6403df411be6
+size 6241217904

model-00027-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:332260d4dd6c96a68cc679c712032caa5dda9ed6170f0b38b98917428ca1aa7d
+size 6241217904

model-00028-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d67a39cce185e1c5ae88efe66247b94231de6ad4dd43d9696fd2a517a56a0ea2
+size 6241217904

model-00029-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:300e136169bba055b66c2c47602494a4457e2c3f06764f2b7ba17cbaa9599e7f
+size 6241217904

model-00030-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6dd9ec5fdd4b8bb609de7cc6c6dc3b5b18af98df086d6bc3c2da678f6d81fe78
+size 6241217904

model-00031-of-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c133427faaa6fc619549824fd96f60b2a1901eca7a0c47c85f8021a6168c3a7
+size 6241217904

modeling_llada2_moe.py

@@ -0,0 +1,1800 @@
+# Copyright 2025 Antgroup and The HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
+# and OPT implementations in this library. It has been modified from its
+# original forms to accommodate minor architectural differences compared
+# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch LLaDA2MoE model."""
+
+import math
+from typing import List, Callable, Optional, Tuple, Union
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+from torch.nn import CrossEntropyLoss
+
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.masking_utils import create_bidirectional_mask
+from transformers.modeling_outputs import (
+    MoeModelOutputWithPast,
+    MoeCausalLMOutputWithPast,
+)
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from transformers.processing_utils import Unpack
+from transformers.pytorch_utils import (
+    ALL_LAYERNORM_LAYERS,
+)
+from transformers.utils import (
+    TransformersKwargs,
+    add_start_docstrings,
+    add_start_docstrings_to_model_forward,
+    logging,
+    replace_return_docstrings,
+)
+from .configuration_llada2_moe import LLaDA2MoeConfig
+from transformers.generation.utils import GenerationMixin
+
+
+logger = logging.get_logger(__name__)
+
+_CONFIG_FOR_DOC = "LLaDA2MoeConfig"
+
+
+def _get_unpad_data(attention_mask):
+    seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
+    indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
+    max_seqlen_in_batch = seqlens_in_batch.max().item()
+    cu_seqlens = F.pad(
+        torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.torch.int32), (1, 0)
+    )
+    return (
+        indices,
+        cu_seqlens,
+        max_seqlen_in_batch,
+    )
+
+
+class LLaDA2MoeRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        LLaDA2MoeRMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+
+ALL_LAYERNORM_LAYERS.append(LLaDA2MoeRMSNorm)
+
+
+class LLaDA2MoeRotaryEmbedding(nn.Module):
+    inv_freq: torch.Tensor  # fix linting for register_buffer
+
+    def __init__(self, config: LLaDA2MoeConfig, device=None):
+        super().__init__()
+        self.max_seq_len_cached = config.max_position_embeddings
+        self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+        
+        self.rope_type = self.config.rope_parameters["rope_type"]
+        rope_init_fn: Callable = self.compute_default_rope_parameters
+        if self.rope_type != "default":
+            rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+        inv_freq, self.attention_scaling = rope_init_fn(self.config, device)
+
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.register_buffer("original_inv_freq", inv_freq.clone(), persistent=False)
+    
+    @staticmethod
+    def compute_default_rope_parameters(
+        config: LLaDA2MoeConfig = None,
+        device=None,
+        seq_len: int = None,
+    ):
+        base = config.rope_parameters["rope_theta"]
+        partial_rotary_factor = config.rope_parameters.get("partial_rotary_factor", 1.0)
+        head_dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
+        dim = int(head_dim * partial_rotary_factor)
+
+        attention_factor = 1.0  # Unused in this type of RoPE
+
+        inv_freq = 1.0 / (
+            base ** (torch.arange(0, dim, 2, dtype=torch.int64).to(device=device, dtype=torch.float) / dim)
+        )
+        return inv_freq, attention_factor
+
+    @torch.no_grad()
+    @dynamic_rope_update  # power user: used with advanced RoPE types (e.g. dynamic rope)
+    def forward(self, x, position_ids):
+        inv_freq_expanded = (
+            self.inv_freq[None, :, None]
+            .float()
+            .expand(position_ids.shape[0], -1, 1)
+            .to(x.device)
+        )
+        position_ids_expanded = position_ids[:, None, :].float()
+
+        device_type = (
+            x.device.type
+            if isinstance(x.device.type, str) and x.device.type != "mps"
+            else "cpu"
+        )
+        with torch.autocast(device_type=device_type, enabled=False):  # Force float32
+            freqs = (
+                inv_freq_expanded.float() @ position_ids_expanded.float()
+            ).transpose(1, 2)
+            emb = torch.cat((freqs, freqs), dim=-1)
+            cos = emb.cos() * self.attention_scaling
+            sin = emb.sin() * self.attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+# Copied from transformers.models.llama.modeling_llama.rotate_half
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+# Copied from transformers.models.llama.modeling_llama.apply_rotary_pos_emb
+def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+        position_ids (`torch.Tensor`):
+            The position indices of the tokens corresponding to the query and key tensors. For example, this can be
+            used to pass offsetted position ids when working with a KV-cache.
+        unsqueeze_dim (`int`, *optional*, defaults to 1):
+            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
+            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
+            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
+            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
+            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
+            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
+    Returns:
+        `tuple(torch.Tensor)` comprising the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos = cos.unsqueeze(unsqueeze_dim)
+    sin = sin.unsqueeze(unsqueeze_dim)
+
+    # Keep half or full tensor for later concatenation
+    rotary_dim = cos.shape[-1]
+    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
+    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
+
+    # Apply rotary embeddings on the first half or full tensor
+    q_embed = (q_rot * cos) + (rotate_half(q_rot) * sin)
+    k_embed = (k_rot * cos) + (rotate_half(k_rot) * sin)
+
+    # Concatenate back to full shape
+    q_embed = torch.cat([q_embed, q_pass], dim=-1)
+    k_embed = torch.cat([k_embed, k_pass], dim=-1)
+    return q_embed, k_embed
+
+
+class LLaDA2MoeMLP(nn.Module):
+    def __init__(self, config: LLaDA2MoeConfig, intermediate_size: int):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = intermediate_size
+
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.act_fn = ACT2FN[config.hidden_act]
+
+    def forward(self, x):
+        return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+
+
+class LLaDA2MoeGate(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.top_k = config.num_experts_per_tok
+        self.num_experts = config.num_experts
+
+        # Block routing
+        self.block_size = config.block_size
+        self.expert_capacity = config.expert_capacity
+
+        # topk selection algorithm
+        self.gating_dim = config.hidden_size
+        self.weight = nn.Parameter(torch.empty((self.num_experts, self.gating_dim)))
+        self.routed_scaling_factor = config.routed_scaling_factor
+
+        self.register_buffer("expert_bias", torch.zeros(self.num_experts))
+        self.reset_parameters()
+
+    def reset_parameters(self) -> None:
+        import torch.nn.init as init
+
+        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
+
+    def block_routing(self, scores_for_routing: torch.Tensor):
+        """Block-level top-k routing: first select expert_capacity experts per block,
+        then do per-token top-k within the allowed set."""
+        num_tokens = scores_for_routing.shape[0]
+        assert num_tokens % self.block_size == 0
+
+        num_blocks = num_tokens // self.block_size
+        # Reshape to (num_blocks, block_size, num_experts)
+        block_routing_scores = scores_for_routing.view(num_blocks, self.block_size, self.num_experts)
+
+        # Phase 1: compute block-level expert scores (max over tokens in each block)
+        block_expert_scores = block_routing_scores.max(dim=1).values  # (num_blocks, num_experts)
+
+        # Select top expert_capacity experts per block
+        _, block_routing_indices = torch.topk(
+            block_expert_scores, k=self.expert_capacity, dim=-1
+        )  # (num_blocks, expert_capacity)
+
+        # Build allowed mask: (num_blocks, num_experts)
+        allowed_mask = torch.zeros(
+            num_blocks, self.num_experts, dtype=torch.bool, device=scores_for_routing.device
+        )
+        allowed_mask.scatter_(1, block_routing_indices, True)
+
+        # Expand mask to per-token level: (num_tokens, num_experts)
+        allowed_mask = allowed_mask.unsqueeze(1).expand(-1, self.block_size, -1).reshape(num_tokens, self.num_experts)
+
+        # Phase 2: per-token top-k within allowed experts
+        masked_scores = scores_for_routing.masked_fill(~allowed_mask, -torch.inf)
+        _, topk_idx = torch.topk(masked_scores, k=self.top_k, dim=-1)
+
+        return topk_idx
+
+    def forward(self, hidden_states):
+        # compute gating score
+        hidden_states = hidden_states.view(-1, hidden_states.shape[-1])
+        logits = F.linear(
+            hidden_states.type(torch.float32), self.weight.type(torch.float32)
+        )
+
+        scores = torch.sigmoid(logits.float()).type_as(logits)
+
+        scores_for_routing = scores + self.expert_bias
+
+        topk_idx = self.block_routing(scores_for_routing)
+
+        scores = torch.gather(scores, dim=1, index=topk_idx).type_as(logits)
+        topk_weight = (
+            scores / (scores.sum(dim=-1, keepdim=True) + 1e-20)
+            if self.top_k > 1
+            else scores
+        )
+        topk_weight = topk_weight * self.routed_scaling_factor
+
+        return topk_idx, topk_weight, logits
+
+
+class LLaDA2MoeSparseMoeBlock(nn.Module):
+    """
+    A mixed expert module containing shared experts.
+    """
+
+    def __init__(self, config: LLaDA2MoeConfig):
+        super().__init__()
+        self.config = config
+        self.num_experts_per_tok = config.num_experts_per_tok
+        self._setup_experts()
+        self.gate = LLaDA2MoeGate(config)
+        if config.num_shared_experts is not None:
+            self.shared_experts = LLaDA2MoeMLP(
+                config=config,
+                intermediate_size=config.moe_intermediate_size
+                * config.num_shared_experts,
+            )
+
+    def _setup_experts(self):
+        self.experts = nn.ModuleList(
+            [
+                LLaDA2MoeMLP(
+                    config=self.config,
+                    intermediate_size=self.config.moe_intermediate_size,
+                )
+                for _ in range(self.config.num_experts)
+            ]
+        )
+
+    def forward(self, hidden_states):
+        identity = hidden_states
+        bsz, seq_len, h = hidden_states.shape
+        topk_idx, topk_weight, router_logits = self.gate(hidden_states)
+        hidden_states = hidden_states.view(-1, hidden_states.shape[-1])
+        flat_topk_idx = topk_idx.view(-1)
+        if self.training:
+            hidden_states = hidden_states.repeat_interleave(
+                self.num_experts_per_tok, dim=0
+            )
+            y = torch.empty_like(hidden_states)
+            for i, expert in enumerate(self.experts):
+                y[flat_topk_idx == i] = expert(hidden_states[flat_topk_idx == i])
+            y = (y.view(*topk_weight.shape, -1) * topk_weight.unsqueeze(-1)).sum(dim=1)
+            y = y.to(hidden_states.dtype).view(bsz, seq_len, h)
+        else:
+            y = self.moe_infer(hidden_states, topk_idx, topk_weight).view(
+                bsz, seq_len, h
+            )
+        if self.config.num_shared_experts is not None:
+            y = y + self.shared_experts(identity)
+        return y, (
+            router_logits.view(bsz, seq_len, -1),
+            topk_idx.view(bsz, seq_len, -1),
+        )
+
+    @torch.no_grad()
+    def moe_infer(self, x, topk_ids, topk_weight):
+        cnts = topk_ids.new_zeros((topk_ids.shape[0], len(self.experts)))
+        cnts.scatter_(1, topk_ids, 1)
+        tokens_per_expert = cnts.sum(dim=0)
+        idxs = topk_ids.view(-1).argsort()
+        sorted_tokens = x[idxs // topk_ids.shape[1]]
+        tokens_per_expert = tokens_per_expert.cpu().numpy()
+        outputs = []
+        start_idx = 0
+        for i, num_tokens_tensor in enumerate(tokens_per_expert):
+            num_tokens = num_tokens_tensor.item()
+            if num_tokens == 0:
+                continue
+            end_idx = start_idx + num_tokens
+            expert = self.experts[i]
+            tokens_for_this_expert = sorted_tokens[start_idx:end_idx]
+            expert_out = expert(tokens_for_this_expert)
+            outputs.append(expert_out.to(x.device))
+            start_idx = end_idx
+
+        outs = torch.cat(outputs, dim=0) if len(outputs) else sorted_tokens.new_empty(0)
+        new_x = torch.empty_like(outs)
+        new_x[idxs] = outs
+        final_out = (
+            new_x.view(*topk_ids.shape, -1)
+            .type(topk_weight.dtype)
+            .mul_(topk_weight.unsqueeze(dim=-1))
+            .sum(dim=1)
+            .type(new_x.dtype)
+        )
+        return final_out
+
+
+# Copied from transformers.models.llama.modeling_llama.repeat_kv
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(
+        batch, num_key_value_heads, n_rep, slen, head_dim
+    )
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: Optional[torch.Tensor],
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs: Unpack[TransformersKwargs],
+):
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        attn_weights = attn_weights + attention_mask[:, :, :, : key_states.shape[-2]]
+
+    # upcast attention to fp32
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(
+        query.dtype
+    )
+    attn_weights = nn.functional.dropout(
+        attn_weights, p=dropout, training=module.training
+    )
+    attn_output = torch.matmul(attn_weights, value_states)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+# Copied from transformers.models.llama.modeling_llama.LlamaAttention with Llama->LLaDA2Moe
+class LLaDA2MoeAttention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config: LLaDA2MoeConfig, layer_idx: Optional[int] = None):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        if layer_idx is None:
+            logger.warning_once(
+                f"Instantiating {self.__class__.__name__} without passing `layer_idx` is not recommended and will "
+                "to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` "
+                "when creating this class."
+            )
+        self.attention_dropout = config.attention_dropout
+        self.hidden_size = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = config.head_dim or self.hidden_size // self.num_heads
+        partial_rotary_factor = (
+            config.partial_rotary_factor
+            if hasattr(config, "partial_rotary_factor")
+            else 1.0
+        )
+        self.rope_dim = int(self.head_dim * partial_rotary_factor)
+        self.num_key_value_heads = config.num_key_value_heads
+        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
+        self.max_position_embeddings = config.max_position_embeddings
+        self.rope_theta = config.rope_theta
+        self.scaling = self.head_dim**-0.5
+        self.is_causal = False
+
+        self.query_key_value = nn.Linear(
+            self.hidden_size,
+            (self.num_heads + 2 * self.num_key_value_heads) * self.head_dim,
+            bias=config.use_qkv_bias,
+        )
+
+        if self.config.use_qk_norm:
+            self.query_layernorm = LLaDA2MoeRMSNorm(
+                self.head_dim, eps=config.rms_norm_eps
+            )
+            self.key_layernorm = LLaDA2MoeRMSNorm(
+                self.head_dim, eps=config.rms_norm_eps
+            )
+        self.dense = nn.Linear(
+            self.num_heads * self.head_dim, self.hidden_size, bias=config.use_bias
+        )
+        self.sliding_window = getattr(config, "sliding_window", None)
+
+    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
+        return (
+            tensor.view(bsz, seq_len, self.num_heads, self.head_dim)
+            .transpose(1, 2)
+            .contiguous()
+        )
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Cache] = None,
+        output_attentions: bool = False,
+        use_cache: bool = False,
+        position_embeddings: Optional[
+            Tuple[torch.Tensor, torch.Tensor]
+        ] = None,  # necessary, but kept here for BC
+        **kwargs,
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
+        input_shape = hidden_states.shape[:-1]
+
+        bsz, q_len, _ = hidden_states.size()
+
+        qkv = self.query_key_value(hidden_states)
+        qkv = qkv.view(
+            bsz, q_len, self.num_heads + 2 * self.num_key_value_heads, self.head_dim
+        )
+
+        query_states, key_states, value_states = qkv.split(
+            [self.num_heads, self.num_key_value_heads, self.num_key_value_heads], dim=-2
+        )
+        query_states = query_states.transpose(1, 2)
+        key_states = key_states.transpose(1, 2)
+        value_states = value_states.transpose(1, 2)
+
+        if self.config.use_qk_norm:
+            query_states = self.query_layernorm(query_states)
+            key_states = self.key_layernorm(key_states)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(
+            query_states, key_states, cos, sin
+        )
+
+        if past_key_value is not None:
+            if self.layer_idx is None:
+                raise ValueError(
+                    f"The cache structure has changed since version v4.36. If you are using {self.__class__.__name__} "
+                    "for auto-regressive decoding with k/v caching, please make sure to initialize the attention class "
+                    "with a layer index."
+                )
+            cache_kwargs = {"sin": sin, "cos": cos}
+            key_states, value_states = past_key_value.update(
+                key_states, value_states, self.layer_idx, cache_kwargs
+            )
+
+        attention_interface: Callable = eager_attention_forward
+        if self.config._attn_implementation != "eager":
+            attention_interface = ALL_ATTENTION_FUNCTIONS[
+                self.config._attn_implementation
+            ]
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scaling,
+            sliding_window=self.sliding_window,  # diff with Llama
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.dense(attn_output)
+
+        return attn_output, attn_weights, past_key_value
+
+
+class LLaDA2MoeDecoderLayer(nn.Module):
+    def __init__(self, config: LLaDA2MoeConfig, layer_idx: int):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+
+        self.attention = LLaDA2MoeAttention(config=config, layer_idx=layer_idx)
+
+        self.mlp = (
+            LLaDA2MoeSparseMoeBlock(config)
+            if (
+                config.num_experts is not None
+                and layer_idx >= config.first_k_dense_replace
+            )
+            else LLaDA2MoeMLP(config=config, intermediate_size=config.intermediate_size)
+        )
+        self.input_layernorm = LLaDA2MoeRMSNorm(
+            config.hidden_size, eps=config.rms_norm_eps
+        )
+        self.post_attention_layernorm = LLaDA2MoeRMSNorm(
+            config.hidden_size, eps=config.rms_norm_eps
+        )
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Tuple[torch.Tensor]] = None,
+        output_attentions: Optional[bool] = False,
+        output_router_logits: Optional[bool] = False,
+        use_cache: Optional[bool] = False,
+        position_embeddings: Optional[
+            Tuple[torch.Tensor, torch.Tensor]
+        ] = None,  # necessary, but kept here for BC
+        **kwargs,
+    ) -> Tuple[
+        torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]
+    ]:
+        """
+        Args:
+            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`
+            attention_mask (`torch.FloatTensor`, *optional*):
+                attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1,
+                query_sequence_length, key_sequence_length)` if default attention is used.
+            position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+                Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
+                config.n_positions - 1]`.
+            past_key_value (`Tuple(torch.FloatTensor)`, *optional*):
+                cached past key and value projection states
+            output_attentions (`bool`, *optional*):
+                Whether to return the attentions tensors of all attention layers. See `attentions` under
+                returned tensors for more detail.
+            output_router_logits (`bool`, *optional*):
+                Whether or not to return the logits of all the routers. They are useful for computing the router loss,
+                and should not be returned during inference.
+            use_cache (`bool`, *optional*):
+                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding
+                (see `past_key_values`).
+        """
+        residual = hidden_states
+
+        hidden_states = self.input_layernorm(hidden_states)
+
+        # Self Attention
+        hidden_states, self_attn_weights, present_key_value = self.attention(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_value=past_key_value,
+            output_attentions=output_attentions,
+            position_embeddings=position_embeddings,
+            use_cache=use_cache,
+        )
+        hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        if isinstance(hidden_states, tuple):
+            hidden_states, router_logits = hidden_states
+        else:
+            router_logits = None
+        hidden_states = residual + hidden_states.to(residual.device)
+
+        outputs = (hidden_states,)
+
+        if output_attentions:
+            outputs += (self_attn_weights,)
+
+        if use_cache:
+            outputs += (present_key_value,)
+
+        if output_router_logits:
+            outputs += (router_logits,)
+
+        return outputs
+
+
+LLADA2MOE_START_DOCSTRING = r"""
+    This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
+    library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
+    etc.)
+
+    This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.
+    Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
+    and behavior.
+
+    Parameters:
+        config ([`LLaDA2MoeConfig`]):
+            Model configuration class with all the parameters of the model. Initializing with a config file does not
+            load the weights associated with the model, only the configuration. Check out the
+            [`~PreTrainedModel.from_pretrained`] method to load the model weights.
+"""
+
+
+@add_start_docstrings(
+    "The bare LLaDA2Moe Model outputting raw hidden-states without any specific head on top.",
+    LLADA2MOE_START_DOCSTRING,
+)
+class LLaDA2MoePreTrainedModel(PreTrainedModel):
+    config_class = LLaDA2MoeConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["LLaDA2MoeDecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn_2 = False
+    _supports_sdpa = True
+    _supports_flex_attn = True
+    _supports_cache_class = True
+
+    @torch.no_grad()
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        std = self.config.initializer_range
+        if isinstance(module, LLaDA2MoeGate):
+            nn.init.normal_(module.weight, mean=0.0, std=std)
+
+
+LLADA2MOE_INPUTS_DOCSTRING = r"""
+    Args:
+        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
+            Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
+            it.
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            [What are input IDs?](../glossary#input-ids)
+        attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
+
+            - 1 for tokens that are **not masked**,
+            - 0 for tokens that are **masked**.
+
+            [What are attention masks?](../glossary#attention-mask)
+
+            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
+            [`PreTrainedTokenizer.__call__`] for details.
+
+            If `past_key_values` is used, optionally only the last `input_ids` have to be input (see
+            `past_key_values`).
+
+            If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`]
+            and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more
+            information on the default strategy.
+
+            - 1 indicates the head is **not masked**,
+            - 0 indicates the head is **masked**.
+        position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
+            config.n_positions - 1]`.
+
+            [What are position IDs?](../glossary#position-ids)
+        past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*):
+            Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention
+            blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values`
+            returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.
+
+            Two formats are allowed:
+            - a [`~cache_utils.Cache`] instance;
+            - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of
+            shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy
+            cache format.
+
+            The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the
+            legacy cache format will be returned.
+
+            If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't
+            have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids`
+            of shape `(batch_size, sequence_length)`.
+        inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
+            Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
+            is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
+            model's internal embedding lookup matrix.
+        use_cache (`bool`, *optional*):
+            If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see
+            `past_key_values`).
+        output_attentions (`bool`, *optional*):
+            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
+            tensors for more detail.
+        output_hidden_states (`bool`, *optional*):
+            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
+            more detail.
+        return_dict (`bool`, *optional*):
+            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
+"""
+
+
+@add_start_docstrings(
+    "The bare LLaDA2Moe Model outputting raw hidden-states without any specific head on top.",
+    LLADA2MOE_START_DOCSTRING,
+)
+class LLaDA2MoeModel(LLaDA2MoePreTrainedModel):
+    """
+    Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LLaDA2MoeDecoderLayer`]
+
+    Args:
+        config: LLaDA2MoeConfig
+    """
+
+    def __init__(self, config: LLaDA2MoeConfig):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.word_embeddings = nn.Embedding(
+            config.vocab_size, config.hidden_size, self.padding_idx
+        )
+        self.layers = nn.ModuleList(
+            [
+                LLaDA2MoeDecoderLayer(config, layer_idx)
+                for layer_idx in range(config.num_hidden_layers)
+            ]
+        )
+        self._use_sdpa = config._attn_implementation == "sdpa"
+        self._use_flex_attention = config._attn_implementation == "flex_attention"
+        self.norm = LLaDA2MoeRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = LLaDA2MoeRotaryEmbedding(config=config)
+        self.gradient_checkpointing = False
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.word_embeddings
+
+    def set_input_embeddings(self, value):
+        self.word_embeddings = value
+
+    @add_start_docstrings_to_model_forward(LLADA2MOE_INPUTS_DOCSTRING)
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_router_logits: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> Union[Tuple, MoeModelOutputWithPast]:
+        output_attentions = (
+            output_attentions
+            if output_attentions is not None
+            else self.config.output_attentions
+        )
+        output_hidden_states = (
+            output_hidden_states
+            if output_hidden_states is not None
+            else self.config.output_hidden_states
+        )
+        output_router_logits = (
+            output_router_logits
+            if output_router_logits is not None
+            else self.config.output_router_logits
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+
+        return_dict = (
+            return_dict if return_dict is not None else self.config.use_return_dict
+        )
+
+        # retrieve input_ids and inputs_embeds
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError(
+                "You cannot specify both input_ids and inputs_embeds at the same time"
+            )
+        elif input_ids is not None:
+            batch_size, seq_length = input_ids.shape[:2]
+        elif inputs_embeds is not None:
+            batch_size, seq_length = inputs_embeds.shape[:2]
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        if self.gradient_checkpointing and self.training:
+            if use_cache:
+                logger.warning_once(
+                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`transformers."
+                )
+                use_cache = False
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache()
+
+        if inputs_embeds is None:
+            inputs_embeds = self.word_embeddings(input_ids)
+
+        past_seen_tokens = (
+            past_key_values.get_seq_length() if past_key_values is not None else 0
+        )
+
+        if position_ids is None:
+            position_ids = torch.arange(
+                past_seen_tokens,
+                past_seen_tokens + inputs_embeds.shape[1],
+                device=inputs_embeds.device,
+            )
+            position_ids = position_ids.unsqueeze(0)
+
+        attention_mask = create_bidirectional_mask(
+            config=self.config,
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+        )
+
+        # embed positions
+        hidden_states = inputs_embeds
+
+        # create position embeddings to be shared across the decoder layers
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        # decoder layers
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attns = () if output_attentions else None
+        all_router_logits = () if output_router_logits else None
+        next_decoder_cache = None
+
+        for decoder_layer in self.layers:
+            if output_hidden_states:
+                all_hidden_states += (hidden_states,)
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    decoder_layer.__call__,
+                    hidden_states,
+                    attention_mask,
+                    position_ids,
+                    past_key_values,
+                    output_attentions,
+                    output_router_logits,
+                    use_cache,
+                    position_embeddings,
+                )
+            else:
+                layer_outputs = decoder_layer(
+                    hidden_states,
+                    attention_mask=attention_mask,
+                    position_ids=position_ids,
+                    past_key_value=past_key_values,
+                    output_attentions=output_attentions,
+                    output_router_logits=output_router_logits,
+                    use_cache=use_cache,
+                    position_embeddings=position_embeddings,
+                )
+            hidden_states = layer_outputs[0]
+
+            if use_cache:
+                next_decoder_cache = layer_outputs[2 if output_attentions else 1]
+
+            if output_attentions:
+                all_self_attns += (layer_outputs[1],)
+
+            if output_router_logits and layer_outputs[-1] is not None:
+                all_router_logits += (layer_outputs[-1],)
+
+        hidden_states = self.norm(hidden_states)
+
+        # add hidden states from the last decoder layer
+        if output_hidden_states:
+            all_hidden_states += (hidden_states,)
+
+        next_cache = None
+        if use_cache:
+            next_cache = next_decoder_cache
+        if not return_dict:
+            return tuple(
+                v
+                for v in [
+                    hidden_states,
+                    next_cache,
+                    all_hidden_states,
+                    all_self_attns,
+                    all_router_logits,
+                ]
+                if v is not None
+            )
+        return MoeModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=next_cache,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attns,
+            router_logits=all_router_logits,
+        )
+
+
+class LLaDA2MoeModelLM(LLaDA2MoePreTrainedModel, GenerationMixin):
+    _tied_weights_keys = ["lm_head.weight"]
+
+    def __init__(self, config: LLaDA2MoeConfig):
+        super().__init__(config)
+        self.model = LLaDA2MoeModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.model.word_embeddings
+
+    def set_input_embeddings(self, value):
+        self.model.word_embeddings = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def set_decoder(self, decoder):
+        self.model = decoder
+
+    def get_decoder(self):
+        return self.model
+
+    @add_start_docstrings_to_model_forward(LLADA2MOE_INPUTS_DOCSTRING)
+    @replace_return_docstrings(
+        output_type=MoeCausalLMOutputWithPast, config_class=_CONFIG_FOR_DOC
+    )
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_router_logits: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> Union[Tuple, MoeCausalLMOutputWithPast]:
+        r"""
+        Args:
+            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+                Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
+                config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
+                (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
+
+        Returns:
+
+        Example:
+
+        ```python
+        >>> from transformers import AutoTokenizer
+
+        >>> model = LLaDA2MoeForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS)
+        >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER)
+
+        >>> prompt = "Hey, are you conscious? Can you talk to me?"
+        >>> inputs = tokenizer(prompt, return_tensors="pt")
+
+        >>> # Generate
+        >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
+        >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+        "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you."
+        ```"""
+        output_attentions = (
+            output_attentions
+            if output_attentions is not None
+            else self.config.output_attentions
+        )
+        output_hidden_states = (
+            output_hidden_states
+            if output_hidden_states is not None
+            else self.config.output_hidden_states
+        )
+        output_router_logits = (
+            output_router_logits
+            if output_router_logits is not None
+            else self.config.output_router_logits
+        )
+        return_dict = (
+            return_dict if return_dict is not None else self.config.use_return_dict
+        )
+        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            output_router_logits=output_router_logits,
+            return_dict=return_dict,
+            **kwargs,
+        )
+
+        loss = None
+        aux_loss = None
+        hidden_states = outputs[0]
+
+        logits = self.lm_head(hidden_states)
+        logits = logits.float()
+
+        if labels is not None:
+            # LLaDA2.0 will use same label position logits
+            shift_logits = logits
+            shift_labels = labels
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss()
+            shift_logits = shift_logits.view(-1, self.config.vocab_size)
+            shift_labels = shift_labels.view(-1)
+            # Enable model parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            if output_router_logits:
+                output = (aux_loss,) + output
+            return (loss,) + output if loss is not None else output
+
+        return MoeCausalLMOutputWithPast(
+            loss=loss,
+            aux_loss=aux_loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+            router_logits=outputs.router_logits,
+        )
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        attention_mask=None,
+        inputs_embeds=None,
+        token_type_ids=None,
+        **kwargs,
+    ):
+        if past_key_values is not None:
+            if isinstance(past_key_values, Cache):
+                cache_length = past_key_values.get_seq_length()
+                past_length = past_key_values.seen_tokens
+                max_cache_length = (
+                    past_key_values.get_max_length()
+                    if hasattr(past_key_values, "get_max_length")
+                    else past_key_values.get_max_cache_shape()
+                )
+            else:
+                cache_length = past_length = past_key_values[0][0].shape[2]
+                max_cache_length = None
+
+            # Keep only the unprocessed tokens:
+            # 1 - If the length of the attention_mask exceeds the length of input_ids, then we are in a setting where
+            # some of the inputs are exclusivelly passed as part of the cache (e.g. when passing input_embeds as input)
+            if (
+                attention_mask is not None
+                and attention_mask.shape[1] > input_ids.shape[1]
+            ):
+                input_ids = input_ids[:, -(attention_mask.shape[1] - past_length) :]
+            # 2 - If the past_length is smaller than input_ids', then input_ids holds all input tokens. We can discard
+            # input_ids based on the past_length.
+            elif past_length < input_ids.shape[1]:
+                input_ids = input_ids[:, past_length:]
+            # 3 - Otherwise (past_length >= input_ids.shape[1]), let's assume input_ids only has unprocessed tokens.
+
+            # If we are about to go beyond the maximum cache length, we need to crop the input attention mask.
+            if (
+                max_cache_length is not None
+                and attention_mask is not None
+                and cache_length + input_ids.shape[1] > max_cache_length
+            ):
+                attention_mask = attention_mask[:, -max_cache_length:]
+
+        position_ids = kwargs.get("position_ids", None)
+        if attention_mask is not None and position_ids is None:
+            # create position_ids on the fly for batch generation
+            position_ids = attention_mask.long().cumsum(-1) - 1
+            position_ids.masked_fill_(attention_mask == 0, 1)
+            if past_key_values:
+                position_ids = position_ids[:, -input_ids.shape[1] :]
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "position_ids": position_ids,
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "attention_mask": attention_mask,
+            }
+        )
+        return model_inputs
+
+    @staticmethod
+    def _reorder_cache(past_key_values, beam_idx):
+        reordered_past = ()
+        for layer_past in past_key_values:
+            reordered_past += (
+                tuple(
+                    past_state.index_select(0, beam_idx.to(past_state.device))
+                    for past_state in layer_past
+                ),
+            )
+        return reordered_past
+
+    @staticmethod
+    def _top_k_logits(logits, k):
+        if k is None or k <= 0:
+            return logits
+        else:
+            values, _ = torch.topk(logits, k)
+            min_values = values[..., -1, None]
+            return torch.where(
+                logits < min_values, torch.full_like(logits, float("-inf")), logits
+            )
+
+    @staticmethod
+    def _top_p_logits(logits, p):
+        if p is None or p >= 1.0:
+            return logits
+        sorted_logits, sorted_indices = torch.sort(logits, descending=True)
+        cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
+        sorted_mask = cumulative_probs > p
+        sorted_mask[..., 1:] = sorted_mask[..., :-1].clone()
+        sorted_mask[..., 0] = False
+        mask_indices = torch.scatter(
+            torch.full_like(logits, False, dtype=torch.bool),
+            -1,
+            sorted_indices,
+            sorted_mask,
+        )
+        return logits.masked_fill(mask_indices, float("-inf"))
+
+    def _sample_with_temperature_topk_topp(
+        self, logits, temperature=1.0, top_k=0, top_p=1.0
+    ):
+        orig_shape = logits.shape[:-1]
+        vocab_size = logits.shape[-1]
+        logits = logits.reshape(-1, vocab_size)
+        if temperature > 0 and temperature != 1.0:
+            logits = logits / temperature
+        logits = self._top_k_logits(logits, top_k)
+        logits = self._top_p_logits(logits, top_p)
+        probs = F.softmax(logits, dim=-1)
+        token = torch.multinomial(probs, num_samples=1)
+        token_prob = torch.gather(probs, -1, token)
+        return token.view(*orig_shape), token_prob.view(*orig_shape)
+
+    @staticmethod
+    def _get_num_transfer_tokens(block_length, steps):
+        if steps == 0:
+            return torch.tensor([], dtype=torch.int64)
+        base = block_length // steps
+        remainder = block_length % steps
+        num_transfer_tokens = torch.full((steps,), base, dtype=torch.int64)
+        num_transfer_tokens[:remainder] += 1
+        return num_transfer_tokens
+
+    @staticmethod
+    def _apply_edit_operations_with_tracking(
+        block_tokens,
+        old_block_tokens,
+        is_original_mask_snapshot,
+        mask_id,
+        block_length,
+        delete_token_id,
+        split_token_id,
+    ):
+        """Process DELETE and SPLIT tokens in a block while tracking ``is_original_mask``.
+
+        The block is kept at a fixed ``block_length`` by truncating (when SPLIT grows it) or
+        right-padding with masks (when DELETE shrinks it).
+
+        Args:
+            block_tokens: List of token ids in the block (current state after writes).
+            old_block_tokens: List of token ids before this step (used to restore SPLIT's carried token).
+            is_original_mask_snapshot: Per-position bools marking positions that were original masks.
+            mask_id: The mask token id.
+            block_length: Target (fixed) block length.
+            delete_token_id / split_token_id: Special edit-operation token ids.
+
+        Behaviour:
+            - DELETE: skip the token (and its tracking entry).
+            - SPLIT:  token -> [mask_id, old_token], tracking -> [False, snapshot_val].
+            - Kept token: carried through with its snapshot tracking value.
+            - Padding masks (from shrink) are tracked as non-original (False).
+
+        Returns:
+            (result_tokens, result_tracking): both lists of length ``block_length``.
+        """
+        result_tokens = []
+        result_tracking = []
+        for i, token in enumerate(block_tokens):
+            if token == delete_token_id:
+                continue
+            elif token == split_token_id:
+                old_token = old_block_tokens[i] if i < len(old_block_tokens) else mask_id
+                snapshot_val = (
+                    is_original_mask_snapshot[i] if i < len(is_original_mask_snapshot) else False
+                )
+                result_tokens.extend([mask_id, old_token])
+                result_tracking.extend([False, snapshot_val])
+            else:
+                snapshot_val = (
+                    is_original_mask_snapshot[i] if i < len(is_original_mask_snapshot) else False
+                )
+                result_tokens.append(token)
+                result_tracking.append(snapshot_val)
+
+        if len(result_tokens) > block_length:
+            result_tokens = result_tokens[:block_length]
+            result_tracking = result_tracking[:block_length]
+        elif len(result_tokens) < block_length:
+            pad_count = block_length - len(result_tokens)
+            result_tokens.extend([mask_id] * pad_count)
+            result_tracking.extend([False] * pad_count)
+
+        return result_tokens, result_tracking
+
+    def _diffusion_sample(self, logits, temperature=1.0, top_k=None, top_p=None):
+        """Sample a token id per position and return its (raw-softmax) confidence.
+
+        With ``temperature == 0`` this is greedy (argmax). Otherwise it applies
+        temperature / top-k / top-p filtering and samples. The returned probability is always
+        taken from the *unfiltered* softmax so it can be used directly as a confidence score.
+        """
+        orig_shape = logits.shape[:-1]
+        vocab_size = logits.shape[-1]
+        logits = logits.reshape(-1, vocab_size)
+        probs_full = F.softmax(logits, dim=-1)
+
+        if temperature is None or temperature == 0.0:
+            token = torch.argmax(logits, dim=-1, keepdim=True)
+        else:
+            scaled = logits / temperature if temperature != 1.0 else logits
+            scaled = self._top_k_logits(scaled, top_k or 0)
+            scaled = self._top_p_logits(scaled, top_p if top_p is not None else 1.0)
+            probs = F.softmax(scaled, dim=-1)
+            token = torch.multinomial(probs, num_samples=1)
+
+        token_prob = torch.gather(probs_full, -1, token)
+        return token.view(*orig_shape), token_prob.view(*orig_shape)
+
+    def _resample_to_escape_loop(
+        self,
+        block_ids,
+        old_block_ids,
+        block_logits,
+        mt2_index,
+        t2t_index,
+        seen_block_results,
+        temperature,
+        top_k,
+        top_p,
+        max_iters=5,
+    ):
+        """Escape a decoding loop by resampling one changed position at a time.
+
+        Called after M2T/T2T writes but BEFORE DELETE/SPLIT processing. If the current
+        (pre-edit) block state repeats a previously seen state, repeatedly pick one random
+        changed position (``block_ids != old_block_ids``) and resample it until the block is
+        novel or ``max_iters`` is reached. The currently chosen token is masked out (-inf)
+        before resampling. M2T positions resample with temperature/top-k/top-p; T2T positions
+        resample greedily. Modifies ``block_ids`` in place.
+        """
+        if tuple(block_ids.tolist()) not in seen_block_results:
+            return
+
+        for _ in range(max_iters):
+            changed_positions = (
+                (block_ids != old_block_ids).nonzero(as_tuple=True)[0].tolist()
+            )
+            if not changed_positions:
+                break
+
+            # Use torch's RNG (not the ``random`` module) so the choice honors torch.manual_seed.
+            rand_idx = torch.randint(len(changed_positions), (1,), device=block_ids.device).item()
+            pos = changed_positions[rand_idx]
+            current_token = block_ids[pos].item()
+            pos_logits = block_logits[pos, :].clone()
+            pos_logits[current_token] = float("-inf")
+
+            if bool(mt2_index[pos]):
+                new_token, _ = self._diffusion_sample(
+                    pos_logits.unsqueeze(0), temperature=temperature, top_k=top_k, top_p=top_p
+                )
+            elif bool(t2t_index[pos]):
+                new_token, _ = self._diffusion_sample(
+                    pos_logits.unsqueeze(0), temperature=0.0, top_k=None, top_p=None
+                )
+            else:
+                # A changed position must be M2T or T2T; skip anything unexpected.
+                continue
+
+            block_ids[pos] = new_token.view(-1)[0]
+            if tuple(block_ids.tolist()) not in seen_block_results:
+                break
+
+    @torch.no_grad()
+    def _joint_decode_block(
+        self,
+        x,
+        block_start,
+        block_end,
+        attention_mask,
+        position_ids,
+        temperature,
+        top_k,
+        top_p,
+        steps,
+        threshold,
+        editing_threshold,
+        max_post_steps,
+        mask_id,
+        delete_token_id,
+        split_token_id,
+        max_steps_per_block,
+    ):
+        """Iteratively refine a single block in place using joint M2T + T2T + edit ops.
+
+        The active block is ``x[0, block_start:block_end]`` (fixed length). Earlier positions
+        of ``x`` provide frozen context via ``attention_mask`` / ``position_ids``. Each step:
+          1. forward over ``x[:, :block_end]``,
+          2. M2T: fill masks whose greedy confidence clears ``threshold`` (with a per-step floor
+             from the ``steps`` transfer schedule),
+          3. T2T: rewrite already-generated tokens whose greedy confidence clears
+             ``editing_threshold`` and whose greedy token differs,
+          4. anti-loop resample if the block state repeats,
+          5. consume DELETE/SPLIT edit tokens (block stays fixed length).
+        Once all *original* masks are gone, up to ``max_post_steps`` further refinement steps
+        run; on the final such step DELETE/SPLIT are suppressed so the block can terminate.
+        """
+        device = x.device
+        block_length = block_end - block_start
+
+        # Positions that were NOT masks at block entry are prompt/context tokens: they are never
+        # written and never carry DELETE/SPLIT, so this mask stays aligned across edits.
+        prompt_mask_block = (x[0, block_start:block_end] != mask_id).clone()
+
+        # A prompt/context position normally never holds a reserved edit token. If one does (bad
+        # upstream template, history, or malformed input) it would be silently deleted/expanded by
+        # the edit-op pass. Warn instead of failing silently.
+        block_ids = x[0, block_start:block_end]
+        prompt_edit_tokens = prompt_mask_block & (
+            (block_ids == delete_token_id) | (block_ids == split_token_id)
+        )
+        if prompt_edit_tokens.any():
+            logger.warning_once(
+                "Reserved edit token(s) found in a prompt/context segment of block@%d; "
+                "they will be deleted/expanded by the edit-op pass.",
+                block_start,
+            )
+
+        is_original_mask = (x[0, block_start:block_end] == mask_id).tolist()
+        initial_mask_count = sum(is_original_mask)
+        if initial_mask_count == 0:
+            return  # fully prompt/context block, nothing to decode
+
+        # Per-step floor for M2T: spread the block's initial masks over ``steps`` steps. Fewer
+        # steps -> more forced unmaskings per step. The schedule sums to ``initial_mask_count``.
+        transfer_schedule = self._get_num_transfer_tokens(initial_mask_count, steps)
+        seen_block_results = set()  # pre-edit block states seen this block, for loop detection
+        post_steps = 0
+        step_id = 0
+
+        while True:
+            blk = x[0, block_start:block_end]
+            old_block = blk.clone()
+            input_key = tuple(old_block.tolist())
+            mask_index = (old_block == mask_id) & (~prompt_mask_block)
+
+            original_mask_count = sum(
+                1
+                for i, v in enumerate(is_original_mask)
+                if v and old_block[i].item() == mask_id
+            )
+            new_mask_count = mask_index.sum().item() - original_mask_count
+
+            # Track post-mask refinement steps: reset while original masks remain.
+            if original_mask_count == 0:
+                post_steps += 1
+            else:
+                post_steps = 0
+
+            logger.debug(
+                "block@%d step=%d original_masks=%d new_masks=%d post_steps=%d/%d",
+                block_start,
+                step_id,
+                original_mask_count,
+                new_mask_count,
+                post_steps,
+                max_post_steps,
+            )
+
+            # Exit guard: require ZERO remaining masks (original *and* new), not just
+            # original_mask_count == 0. An edit op can leave a fresh mask exactly when the last
+            # original mask is resolved; keying off original masks alone would exit early and
+            # return residual mask_id. The final round (below) force-resolves all masks, so this
+            # normally holds immediately; the total-count guard is the belt-and-suspenders check.
+            if mask_index.sum().item() == 0 and post_steps > max_post_steps:
+                logger.debug(
+                    "block@%d terminating: max_post_steps (%d) exceeded after %d steps",
+                    block_start,
+                    max_post_steps,
+                    step_id,
+                )
+                break
+            if step_id >= max_steps_per_block:
+                logger.debug(
+                    "block@%d terminating: max_steps_per_block (%d) reached",
+                    block_start,
+                    max_steps_per_block,
+                )
+                break
+
+            # 1. Forward pass over the current window.
+            logits = self.forward(
+                x[:, :block_end],
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+            ).logits
+            block_logits = logits[0, block_start:block_end, :]
+
+            # 2. Sampled (temperature) result and greedy result + confidences.
+            x_s, p_s = self._diffusion_sample(block_logits, temperature, top_k, top_p)
+            if temperature != 0.0:
+                x0, p0 = self._diffusion_sample(block_logits, 0.0, None, None)
+            else:
+                x0, p0 = x_s, p_s
+
+            neg_inf = torch.full_like(p0, -float("inf"))
+
+            # 3. M2T (mask -> token): threshold-gated with a per-step floor.
+            # The gate uses ``p_s`` -- the confidence of the token that will actually be written
+            # (``x_s``) -- so a position is only unmasked when the sampled token itself is
+            # confident. When temperature == 0, ``p_s == p0``, so this reduces to greedy behavior.
+            mt2_index = torch.zeros(block_length, dtype=torch.bool, device=device)
+            if mask_index.any():
+                if step_id < len(transfer_schedule):
+                    num_need = transfer_schedule[step_id].item() + new_mask_count
+                    mask_conf = torch.where(mask_index, p_s, neg_inf)
+                    high_conf = (mask_conf > threshold) & mask_index
+                    if high_conf.sum().item() >= num_need:
+                        mt2_index = high_conf
+                    else:
+                        k_val = min(num_need, mask_index.sum().item())
+                        if k_val > 0:
+                            _, idx = torch.topk(mask_conf, k=k_val)
+                            mt2_index[idx] = True
+                else:
+                    mt2_index = mask_index
+
+            # 4. T2T (token -> token edit): high-confidence rewrites of generated tokens.
+            editable_mask = (~mask_index) & (~prompt_mask_block)
+            editing_confidence = torch.where(editable_mask, p0, neg_inf)
+            high_conf_edit = (editing_confidence > editing_threshold) & editable_mask
+            token_changed = old_block != x0
+            t2t_index = high_conf_edit & token_changed
+
+            fill_index_pre = mt2_index | t2t_index
+
+            # Final round = the last refinement step, after which the block terminates. On it we
+            # suppress SPLIT/DELETE (below) so no new masks appear.
+            #
+            # max_post_steps > 0: count-based -- fire once we've spent the post-mask step budget.
+            # max_post_steps == 0: "no post steps" means the step that resolves the LAST original
+            #   mask must itself finish the block. post_steps can't detect this (it is still 0 at
+            #   the top of that step), so we detect it from mt2_index: this step is final iff its
+            #   M2T covers every still-unresolved original mask. Guaranteed to fire eventually --
+            #   the transfer schedule forces mt2_index == mask_index by step len(transfer_schedule).
+            if max_post_steps > 0:
+                final_round = post_steps >= max_post_steps
+            else:
+                remaining_original = mask_index & torch.tensor(is_original_mask, device=device)
+                final_round = remaining_original.any() and (remaining_original <= mt2_index).all()
+
+            # 5. On the final round, suppress SPLIT/DELETE so the block can terminate. Only the
+            # positions that are actually written matter: M2T writes x_s, T2T writes x0. Other
+            # positions may still contain S/D harmlessly since they are never written. Resampling
+            # is batched over all offending positions (S/D columns masked to -inf).
+            if final_round:
+                # The final round must leave the block fully unmasked, so resolve EVERY remaining
+                # mask here -- before the D/S suppression below, so these positions also get their
+                # SPLIT/DELETE stripped and no fresh mask survives. For max_post_steps > 0 this is
+                # already implied (once original masks are gone, new_mask_count inflates num_need so
+                # M2T selects all masks anyway), so it is a no-op there; stating it makes the
+                # invariant explicit and robust to changes in the M2T selection above.
+                mt2_index = mask_index
+                fill_index_pre = mt2_index | t2t_index
+
+                m2t_sd = mt2_index & ((x_s == split_token_id) | (x_s == delete_token_id))
+                if m2t_sd.any():
+                    sd_logits = block_logits[m2t_sd].clone()
+                    sd_logits[:, split_token_id] = float("-inf")
+                    sd_logits[:, delete_token_id] = float("-inf")
+                    new_tokens, _ = self._diffusion_sample(
+                        sd_logits, temperature, top_k, top_p
+                    )
+                    x_s[m2t_sd] = new_tokens
+
+                t2t_sd = t2t_index & ((x0 == split_token_id) | (x0 == delete_token_id))
+                if t2t_sd.any():
+                    sd_logits = block_logits[t2t_sd].clone()
+                    sd_logits[:, split_token_id] = float("-inf")
+                    sd_logits[:, delete_token_id] = float("-inf")
+                    new_tokens, _ = self._diffusion_sample(
+                        sd_logits, temperature=0.0, top_k=None, top_p=None
+                    )
+                    x0[t2t_sd] = new_tokens
+
+            # 6. Apply writes: M2T writes the sampled token, T2T writes the greedy token.
+            is_original_mask_snapshot = list(is_original_mask)
+            if fill_index_pre.any():
+                if mt2_index.any():
+                    blk[mt2_index] = x_s[mt2_index]
+                if t2t_index.any():
+                    blk[t2t_index] = x0[t2t_index]
+
+                # Anti-loop: escape a repeated pre-edit block state. Skipped on the final round,
+                # where SD suppression already drives termination.
+                if not final_round:
+                    self._resample_to_escape_loop(
+                        blk,
+                        old_block,
+                        block_logits,
+                        mt2_index,
+                        t2t_index,
+                        seen_block_results,
+                        temperature=temperature,
+                        top_k=top_k,
+                        top_p=top_p,
+                    )
+
+            seen_block_results.add(tuple(blk.tolist()))
+
+            # 7. Consume DELETE/SPLIT edit tokens (block stays fixed length).
+            edited_block, is_original_mask = self._apply_edit_operations_with_tracking(
+                blk.tolist(),
+                old_block.tolist(),
+                is_original_mask_snapshot,
+                mask_id,
+                block_length,
+                delete_token_id=delete_token_id,
+                split_token_id=split_token_id,
+            )
+            x[0, block_start:block_end] = torch.tensor(
+                edited_block, device=device, dtype=x.dtype
+            )
+
+            # 8. Stop when the block is stable and fully unmasked.
+            if x[0, block_start:block_end].tolist() == list(input_key) and (
+                x[0, block_start:block_end] == mask_id
+            ).sum() == 0:
+                step_id += 1
+                break
+
+            step_id += 1
+
+    @torch.no_grad()
+    def generate(
+        self,
+        inputs: Optional[torch.Tensor] = None,
+        temperature: float = 0.0,
+        block_length: int = 32,
+        steps: int = 32,
+        gen_length: int = 2048,
+        top_p: Optional[float] = None,
+        top_k: Optional[int] = None,
+        threshold: float = 0.5,
+        editing_threshold: float = 0.0,
+        max_post_steps: int = 16,
+        eos_early_stop: bool = False,
+        eos_id: int = 156892,
+        mask_id: int = 156895,
+        delete_token_id: int = 156930,
+        split_token_id: int = 156931,
+        max_steps_per_block: int = 1000,
+    ):
+        r"""
+        Generate tokens with a block-wise, edit-based iterative refinement strategy.
+
+        Unlike autoregressive generation, this method lays out a full masked template and
+        refines it block by block. Within each block it jointly performs:
+
+        - **M2T** (mask -> token): converts ``mask_id`` placeholders into concrete tokens once
+          their confidence exceeds ``threshold`` (with a per-step floor so progress is
+          guaranteed).
+        - **T2T** (token -> token): rewrites already-generated tokens whose greedy confidence
+          exceeds ``editing_threshold`` and whose greedy prediction differs from the current
+          token.
+        - **DELETE / SPLIT**: consumes special edit tokens to remove positions or insert new
+          masks, letting the block change its content length (kept fixed by pad/truncate).
+
+        An anti-loop resampler perturbs the block whenever a pre-edit state repeats. After all
+        original masks in a block are resolved, up to ``max_post_steps`` further refinement
+        steps run; the final one suppresses DELETE/SPLIT so the block terminates.
+
+        A block-diagonal causal attention mask lets a block attend to all previous blocks (and
+        bidirectionally within itself) but not to future blocks.
+
+        Parameters:
+            inputs (`torch.Tensor`): Prompt token ids of shape ``(1, prompt_length)``.
+            temperature (`float`, defaults to 0.0): 0.0 is greedy; >0 enables sampling for M2T.
+            block_length (`int`, defaults to 32): Fixed length of each generation block.
+            steps (`int`, defaults to 32): Number of steps the M2T transfer schedule spreads a
+                block's initial masks over (per-step unmasking floor). Fewer steps forces more
+                unmaskings per step. Independent of the actual number of iterations, which is
+                driven by the confidence thresholds and the post-mask refinement phase.
+            gen_length (`int`, defaults to 2048): Number of tokens to generate after the prompt.
+            top_p / top_k (`float`/`int`, *optional*): Nucleus / top-k filtering for sampling.
+            threshold (`float`, defaults to 0.5): Confidence threshold for M2T unmasking.
+            editing_threshold (`float`, defaults to 0.0): Confidence threshold for T2T edits.
+            max_post_steps (`int`, defaults to 16): Max refinement steps after a block's original
+                masks are all resolved.
+            eos_early_stop (`bool`, defaults to False): Stop after a block that produced ``eos_id``.
+            eos_id / mask_id (`int`): End-of-sequence and mask placeholder token ids.
+            delete_token_id / split_token_id (`int`): Special edit-operation token ids.
+            max_steps_per_block (`int`, defaults to 1000): Hard safety cap on steps per block.
+
+        Return:
+            `torch.Tensor`: The generated token ids after the prompt, up to and including the
+            first ``eos_id`` (or ``gen_length`` if none is produced).
+        """
+        input_ids = inputs.to(self.device)
+
+        prompt_length = input_ids.shape[1]
+        num_blocks = (prompt_length + gen_length + block_length - 1) // block_length
+        total_length = num_blocks * block_length
+
+        block_mask = torch.tril(torch.ones(num_blocks, num_blocks, device=self.device))
+        block_diffusion_attention_mask = (
+            (
+                block_mask.repeat_interleave(block_length, dim=0)
+                .repeat_interleave(block_length, dim=1)
+                .unsqueeze(0)
+                .unsqueeze(0)
+            )
+            .log()
+            .to(torch.bfloat16)
+        )
+
+        position_ids = torch.arange(total_length, device=self.device).unsqueeze(0)
+        x = torch.full((1, total_length), mask_id, dtype=torch.long, device=self.device)
+        x[:, :prompt_length] = input_ids.clone()
+
+        prefill_blocks = prompt_length // block_length
+
+        for num_block in range(prefill_blocks, num_blocks):
+            block_start = num_block * block_length
+            block_end = (num_block + 1) * block_length
+            cur_attn_mask = block_diffusion_attention_mask[
+                :, :, :block_end, :block_end
+            ]
+            cur_position_ids = position_ids[:, :block_end]
+
+            self._joint_decode_block(
+                x,
+                block_start,
+                block_end,
+                cur_attn_mask,
+                cur_position_ids,
+                temperature=temperature,
+                top_k=top_k,
+                top_p=top_p,
+                steps=steps,
+                threshold=threshold,
+                editing_threshold=editing_threshold,
+                max_post_steps=max_post_steps,
+                mask_id=mask_id,
+                delete_token_id=delete_token_id,
+                split_token_id=split_token_id,
+                max_steps_per_block=max_steps_per_block,
+            )
+
+            if (
+                eos_early_stop
+                and eos_id is not None
+                and (x[0, prompt_length:block_end] == eos_id).any()
+            ):
+                break
+
+        generated_answer = x[:, : prompt_length + gen_length]
+
+        eos_positions = (generated_answer[0][prompt_length:] == eos_id).nonzero(
+            as_tuple=True
+        )[0]
+        if len(eos_positions) > 0:
+            first_eos_position = eos_positions[0].item()
+        else:
+            first_eos_position = gen_length
+        output = generated_answer[
+            :, prompt_length : prompt_length + first_eos_position + 1
+        ]
+
+        # Safety net: a well-formed decode leaves no mask / edit tokens in the output. If any
+        # survive (e.g. a degenerate config such as max_post_steps=0 that fails to converge, or a
+        # block that hit max_steps_per_block), warn instead of silently returning residual
+        # mask_id / split / delete tokens to the caller.
+        residual = (
+            (output == mask_id)
+            | (output == split_token_id)
+            | (output == delete_token_id)
+        )
+        if residual.any():
+            logger.warning(
+                "Decoding finished with %d residual mask/split/delete token(s) in the output; "
+                "the generation may be malformed.",
+                int(residual.sum().item()),
+            )
+
+        return output

special_tokens_map.json

@@ -1,4006 +1,8 @@
 {
-  "additional_special_tokens": [
-    "<unk>",
-    "<|startoftext|>",
-    "<|endoftext|>",
-    "<|fim_prefix|>",
-    "<|fim_middle|>",
-    "<|fim_suffix|>",
-    "<|special_6|>",
-    "<|special_7|>",
-    "<|special_8|>",
-    "<|special_9|>",
-    "<|special_10|>",
-    "<|special_11|>",
-    "<|special_12|>",
-    "<|special_13|>",
-    "<|special_14|>",
-    "<|special_15|>",
-    "<|special_16|>",
-    "<|special_17|>",
-    "<|special_18|>",
-    "<|special_19|>",
-    "<|flush|>",
-    "<|calls|>",
-    "<|tools:begin|>",
-    "<|tools:end|>",
-    "<|tool:begin|>",
-    "<|tool:end|>",
-    "<|tool_response|>",
-    "<|tool_response:begin|>",
-    "<|tool_response:end|>",
-    "<|tool_response:name|>",
-    "<|tool_response:result|>",
-    "<|special_40|>",
-    "<|special_41|>",
-    "<|special_42|>",
-    "<|special_43|>",
-    "<|special_44|>",
-    "<|special_45|>",
-    "<|special_46|>",
-    "<|special_47|>",
-    "<|special_48|>",
-    "<|special_49|>",
-    "<|special_50|>",
-    "<|special_51|>",
-    "<|special_52|>",
-    "<|special_53|>",
-    "<|special_54|>",
-    "<|special_55|>",
-    "<|special_56|>",
-    "<|special_57|>",
-    "<|special_58|>",
-    "<|special_59|>",
-    "<|special_60|>",
-    "<|special_61|>",
-    "<|special_62|>",
-    "<|special_63|>",
-    "<|special_64|>",
-    "<|special_65|>",
-    "<|special_66|>",
-    "<|special_67|>",
-    "<|special_68|>",
-    "<|special_69|>",
-    "<|special_70|>",
-    "<|special_71|>",
-    "<|special_72|>",
-    "<|special_73|>",
-    "<|special_74|>",
-    "<|special_75|>",
-    "<|special_76|>",
-    "<|special_77|>",
-    "<|special_78|>",
-    "<|special_79|>",
-    "<|special_80|>",
-    "<|special_81|>",
-    "<|special_82|>",
-    "<|special_83|>",
-    "<|special_84|>",
-    "<|special_85|>",
-    "<|special_86|>",
-    "<|special_87|>",
-    "<|special_88|>",
-    "<|special_89|>",
-    "<|special_90|>",
-    "<|special_91|>",
-    "<|special_92|>",
-    "<|special_93|>",
-    "<|special_94|>",
-    "<|special_95|>",
-    "<|special_96|>",
-    "<|special_97|>",
-    "<|special_98|>",
-    "<|special_99|>",
-    "<|special_100|>",
-    "<|special_101|>",
-    "<|special_102|>",
-    "<|special_103|>",
-    "<|special_104|>",
-    "<|special_105|>",
-    "<|special_106|>",
-    "<|special_107|>",
-    "<|special_108|>",
-    "<|special_109|>",
-    "<|special_110|>",
-    "<|special_111|>",
-    "<|special_112|>",
-    "<|special_113|>",
-    "<|special_114|>",
-    "<|special_115|>",
-    "<|special_116|>",
-    "<|special_117|>",
-    "<|special_118|>",
-    "<|special_119|>",
-    "<|special_120|>",
-    "<|special_121|>",
-    "<|special_122|>",
-    "<|special_123|>",
-    "<|special_124|>",
-    "<|special_125|>",
-    "<|special_126|>",
-    "<|special_127|>",
-    "<|special_128|>",
-    "<|special_129|>",
-    "<|special_130|>",
-    "<|special_131|>",
-    "<|special_132|>",
-    "<|special_133|>",
-    "<|special_134|>",
-    "<|special_135|>",
-    "<|special_136|>",
-    "<|special_137|>",
-    "<|special_138|>",
-    "<|special_139|>",
-    "<|special_140|>",
-    "<|special_141|>",
-    "<|special_142|>",
-    "<|special_143|>",
-    "<|special_144|>",
-    "<|special_145|>",
-    "<|special_146|>",
-    "<|special_147|>",
-    "<|special_148|>",
-    "<|special_149|>",
-    "<|special_150|>",
-    "<|special_151|>",
-    "<|special_152|>",
-    "<|special_153|>",
-    "<|special_154|>",
-    "<|special_155|>",
-    "<|special_156|>",
-    "<|special_157|>",
-    "<|special_158|>",
-    "<|special_159|>",
-    "<|special_160|>",
-    "<|special_161|>",
-    "<|special_162|>",
-    "<|special_163|>",
-    "<|special_164|>",
-    "<|special_165|>",
-    "<|special_166|>",
-    "<|special_167|>",
-    "<|special_168|>",
-    "<|special_169|>",
-    "<|special_170|>",
-    "<|special_171|>",
-    "<|special_172|>",
-    "<|special_173|>",
-    "<|special_174|>",
-    "<|special_175|>",
-    "<|special_176|>",
-    "<|special_177|>",
-    "<|special_178|>",
-    "<|special_179|>",
-    "<|special_180|>",
-    "<|special_181|>",
-    "<|special_182|>",
-    "<|special_183|>",
-    "<|special_184|>",
-    "<|special_185|>",
-    "<|special_186|>",
-    "<|special_187|>",
-    "<|special_188|>",
-    "<|special_189|>",
-    "<|special_190|>",
-    "<|special_191|>",
-    "<|special_192|>",
-    "<|special_193|>",
-    "<|special_194|>",
-    "<|special_195|>",
-    "<|special_196|>",
-    "<|special_197|>",
-    "<|special_198|>",
-    "<|special_199|>",
-    "<|special_200|>",
-    "<|special_201|>",
-    "<|special_202|>",
-    "<|special_203|>",
-    "<|special_204|>",
-    "<|special_205|>",
-    "<|special_206|>",
-    "<|special_207|>",
-    "<|special_208|>",
-    "<|special_209|>",
-    "<|special_210|>",
-    "<|special_211|>",
-    "<|special_212|>",
-    "<|special_213|>",
-    "<|special_214|>",
-    "<|special_215|>",
-    "<|special_216|>",
-    "<|special_217|>",
-    "<|special_218|>",
-    "<|special_219|>",
-    "<|special_220|>",
-    "<|special_221|>",
-    "<|special_222|>",
-    "<|special_223|>",
-    "<|special_224|>",
-    "<|special_225|>",
-    "<|special_226|>",
-    "<|special_227|>",
-    "<|special_228|>",
-    "<|special_229|>",
-    "<|special_230|>",
-    "<|special_231|>",
-    "<|special_232|>",
-    "<|special_233|>",
-    "<|special_234|>",
-    "<|special_235|>",
-    "<|special_236|>",
-    "<|special_237|>",
-    "<|special_238|>",
-    "<|special_239|>",
-    "<|special_240|>",
-    "<|special_241|>",
-    "<|special_242|>",
-    "<|special_243|>",
-    "<|special_244|>",
-    "<|special_245|>",
-    "<|special_246|>",
-    "<|special_247|>",
-    "<|special_248|>",
-    "<|special_249|>",
-    "<|special_250|>",
-    "<|special_251|>",
-    "<|special_252|>",
-    "<|special_253|>",
-    "<|special_254|>",
-    "<|special_255|>",
-    "<|special_256|>",
-    "<|special_257|>",
-    "<|special_258|>",
-    "<|special_259|>",
-    "<|special_260|>",
-    "<|special_261|>",
-    "<|special_262|>",
-    "<|special_263|>",
-    "<|special_264|>",
-    "<|special_265|>",
-    "<|special_266|>",
-    "<|special_267|>",
-    "<|special_268|>",
-    "<|special_269|>",
-    "<|special_270|>",
-    "<|special_271|>",
-    "<|special_272|>",
-    "<|special_273|>",
-    "<|special_274|>",
-    "<|special_275|>",
-    "<|special_276|>",
-    "<|special_277|>",
-    "<|special_278|>",
-    "<|special_279|>",
-    "<|special_280|>",
-    "<|special_281|>",
-    "<|special_282|>",
-    "<|special_283|>",
-    "<|special_284|>",
-    "<|special_285|>",
-    "<|special_286|>",
-    "<|special_287|>",
-    "<|special_288|>",
-    "<|special_289|>",
-    "<|special_290|>",
-    "<|special_291|>",
-    "<|special_292|>",
-    "<|special_293|>",
-    "<|special_294|>",
-    "<|special_295|>",
-    "<|special_296|>",
-    "<|special_297|>",
-    "<|special_298|>",
-    "<|special_299|>",
-    "<|special_300|>",
-    "<|special_301|>",
-    "<|special_302|>",
-    "<|special_303|>",
-    "<|special_304|>",
-    "<|special_305|>",
-    "<|special_306|>",
-    "<|special_307|>",
-    "<|special_308|>",
-    "<|special_309|>",
-    "<|special_310|>",
-    "<|special_311|>",
-    "<|special_312|>",
-    "<|special_313|>",
-    "<|special_314|>",
-    "<|special_315|>",
-    "<|special_316|>",
-    "<|special_317|>",
-    "<|special_318|>",
-    "<|special_319|>",
-    "<|special_320|>",
-    "<|special_321|>",
-    "<|special_322|>",
-    "<|special_323|>",
-    "<|special_324|>",
-    "<|special_325|>",
-    "<|special_326|>",
-    "<|special_327|>",
-    "<|special_328|>",
-    "<|special_329|>",
-    "<|special_330|>",
-    "<|special_331|>",
-    "<|special_332|>",
-    "<|special_333|>",
-    "<|special_334|>",
-    "<|special_335|>",
-    "<|special_336|>",
-    "<|special_337|>",
-    "<|special_338|>",
-    "<|special_339|>",
-    "<|special_340|>",
-    "<|special_341|>",
-    "<|special_342|>",
-    "<|special_343|>",
-    "<|special_344|>",
-    "<|special_345|>",
-    "<|special_346|>",
-    "<|special_347|>",
-    "<|special_348|>",
-    "<|special_349|>",
-    "<|special_350|>",
-    "<|special_351|>",
-    "<|special_352|>",
-    "<|special_353|>",
-    "<|special_354|>",
-    "<|special_355|>",
-    "<|special_356|>",
-    "<|special_357|>",
-    "<|special_358|>",
-    "<|special_359|>",
-    "<|special_360|>",
-    "<|special_361|>",
-    "<|special_362|>",
-    "<|special_363|>",
-    "<|special_364|>",
-    "<|special_365|>",
-    "<|special_366|>",
-    "<|special_367|>",
-    "<|special_368|>",
-    "<|special_369|>",
-    "<|special_370|>",
-    "<|special_371|>",
-    "<|special_372|>",
-    "<|special_373|>",
-    "<|special_374|>",
-    "<|special_375|>",
-    "<|special_376|>",
-    "<|special_377|>",
-    "<|special_378|>",
-    "<|special_379|>",
-    "<|special_380|>",
-    "<|special_381|>",
-    "<|special_382|>",
-    "<|special_383|>",
-    "<|special_384|>",
-    "<|special_385|>",
-    "<|special_386|>",
-    "<|special_387|>",
-    "<|special_388|>",
-    "<|special_389|>",
-    "<|special_390|>",
-    "<|special_391|>",
-    "<|special_392|>",
-    "<|special_393|>",
-    "<|special_394|>",
-    "<|special_395|>",
-    "<|special_396|>",
-    "<|special_397|>",
-    "<|special_398|>",
-    "<|special_399|>",
-    "<|special_400|>",
-    "<|special_401|>",
-    "<|special_402|>",
-    "<|special_403|>",
-    "<|special_404|>",
-    "<|special_405|>",
-    "<|special_406|>",
-    "<|special_407|>",
-    "<|special_408|>",
-    "<|special_409|>",
-    "<|special_410|>",
-    "<|special_411|>",
-    "<|special_412|>",
-    "<|special_413|>",
-    "<|special_414|>",
-    "<|special_415|>",
-    "<|special_416|>",
-    "<|special_417|>",
-    "<|special_418|>",
-    "<|special_419|>",
-    "<|special_420|>",
-    "<|special_421|>",
-    "<|special_422|>",
-    "<|special_423|>",
-    "<|special_424|>",
-    "<|special_425|>",
-    "<|special_426|>",
-    "<|special_427|>",
-    "<|special_428|>",
-    "<|special_429|>",
-    "<|special_430|>",
-    "<|special_431|>",
-    "<|special_432|>",
-    "<|special_433|>",
-    "<|special_434|>",
-    "<|special_435|>",
-    "<|special_436|>",
-    "<|special_437|>",
-    "<|special_438|>",
-    "<|special_439|>",
-    "<|special_440|>",
-    "<|special_441|>",
-    "<|special_442|>",
-    "<|special_443|>",
-    "<|special_444|>",
-    "<|special_445|>",
-    "<|special_446|>",
-    "<|special_447|>",
-    "<|special_448|>",
-    "<|special_449|>",
-    "<|special_450|>",
-    "<|special_451|>",
-    "<|special_452|>",
-    "<|special_453|>",
-    "<|special_454|>",
-    "<|special_455|>",
-    "<|special_456|>",
-    "<|special_457|>",
-    "<|special_458|>",
-    "<|special_459|>",
-    "<|special_460|>",
-    "<|special_461|>",
-    "<|special_462|>",
-    "<|special_463|>",
-    "<|special_464|>",
-    "<|special_465|>",
-    "<|special_466|>",
-    "<|special_467|>",
-    "<|special_468|>",
-    "<|special_469|>",
-    "<|special_470|>",
-    "<|special_471|>",
-    "<|special_472|>",
-    "<|special_473|>",
-    "<|special_474|>",
-    "<|special_475|>",
-    "<|special_476|>",
-    "<|special_477|>",
-    "<|special_478|>",
-    "<|special_479|>",
-    "<|special_480|>",
-    "<|special_481|>",
-    "<|special_482|>",
-    "<|special_483|>",
-    "<|special_484|>",
-    "<|special_485|>",
-    "<|special_486|>",
-    "<|special_487|>",
-    "<|special_488|>",
-    "<|special_489|>",
-    "<|special_490|>",
-    "<|special_491|>",
-    "<|special_492|>",
-    "<|special_493|>",
-    "<|special_494|>",
-    "<|special_495|>",
-    "<|special_496|>",
-    "<|special_497|>",
-    "<|special_498|>",
-    "<|special_499|>",
-    "<|special_500|>",
-    "<|special_501|>",
-    "<|special_502|>",
-    "<|special_503|>",
-    "<|special_504|>",
-    "<|special_505|>",
-    "<|special_506|>",
-    "<|special_507|>",
-    "<|special_508|>",
-    "<|special_509|>",
-    "<|special_510|>",
-    "<|special_511|>",
-    "<|special_625|>",
-    "<|special_626|>",
-    "<|special_627|>",
-    "<|special_628|>",
-    "<|special_629|>",
-    "<|special_630|>",
-    "<|special_631|>",
-    "<|special_632|>",
-    "<|special_633|>",
-    "<|special_634|>",
-    "<|special_635|>",
-    "<|special_636|>",
-    "<|special_637|>",
-    "<|special_638|>",
-    "<|special_639|>",
-    "<|special_640|>",
-    "<|special_641|>",
-    "<|special_642|>",
-    "<|special_643|>",
-    "<|special_644|>",
-    "<|special_645|>",
-    "<|special_646|>",
-    "<|special_647|>",
-    "<|special_648|>",
-    "<|special_649|>",
-    "<|special_650|>",
-    "<|special_651|>",
-    "<|special_652|>",
-    "<|special_653|>",
-    "<|special_654|>",
-    "<|special_655|>",
-    "<|special_656|>",
-    "<|special_657|>",
-    "<|special_658|>",
-    "<|special_659|>",
-    "<|special_660|>",
-    "<|special_661|>",
-    "<|special_662|>",
-    "<|special_663|>",
-    "<|special_664|>",
-    "<|special_665|>",
-    "<|special_666|>",
-    "<|special_667|>",
-    "<|special_668|>",
-    "<|special_669|>",
-    "<|special_670|>",
-    "<|special_671|>",
-    "<|special_672|>",
-    "<|special_673|>",
-    "<|special_674|>",
-    "<|special_675|>",
-    "<|special_676|>",
-    "<|special_677|>",
-    "<|special_678|>",
-    "<|special_679|>",
-    "<|special_680|>",
-    "<|special_681|>",
-    "<|special_682|>",
-    "<|special_683|>",
-    "<|special_684|>",
-    "<|special_685|>",
-    "<|special_686|>",
-    "<|special_687|>",
-    "<|special_688|>",
-    "<|special_689|>",
-    "<|special_690|>",
-    "<|special_691|>",
-    "<|special_692|>",
-    "<|special_693|>",
-    "<|special_694|>",
-    "<|special_695|>",
-    "<|special_696|>",
-    "<|special_697|>",
-    "<|special_698|>",
-    "<|special_699|>",
-    "<|special_700|>",
-    "<|special_701|>",
-    "<|special_702|>",
-    "<|special_703|>",
-    "<|special_704|>",
-    "<|special_705|>",
-    "<|special_706|>",
-    "<|special_707|>",
-    "<|special_708|>",
-    "<|special_709|>",
-    "<|special_710|>",
-    "<|special_711|>",
-    "<|special_712|>",
-    "<|special_713|>",
-    "<|special_714|>",
-    "<|special_715|>",
-    "<|special_716|>",
-    "<|special_717|>",
-    "<|special_718|>",
-    "<|special_719|>",
-    "<|special_720|>",
-    "<|special_721|>",
-    "<|special_722|>",
-    "<|special_723|>",
-    "<|special_724|>",
-    "<|special_725|>",
-    "<|special_726|>",
-    "<|special_727|>",
-    "<|special_728|>",
-    "<|special_729|>",
-    "<|special_730|>",
-    "<|special_731|>",
-    "<|special_732|>",
-    "<|special_733|>",
-    "<|special_734|>",
-    "<|special_735|>",
-    "<|special_736|>",
-    "<|special_737|>",
-    "<|special_738|>",
-    "<|special_739|>",
-    "<|special_740|>",
-    "<|special_741|>",
-    "<|special_742|>",
-    "<|special_743|>",
-    "<|special_744|>",
-    "<|special_745|>",
-    "<|special_746|>",
-    "<|special_747|>",
-    "<|special_748|>",
-    "<|special_749|>",
-    "<|special_750|>",
-    "<|special_751|>",
-    "<|special_752|>",
-    "<|special_753|>",
-    "<|special_754|>",
-    "<|special_755|>",
-    "<|special_756|>",
-    "<|special_757|>",
-    "<|special_758|>",
-    "<|special_759|>",
-    "<|special_760|>",
-    "<|special_761|>",
-    "<|special_762|>",
-    "<|special_763|>",
-    "<|special_764|>",
-    "<|special_765|>",
-    "<|special_766|>",
-    "<|special_767|>",
-    "<|special_768|>",
-    "<|special_769|>",
-    "<|special_770|>",
-    "<|special_771|>",
-    "<|special_772|>",
-    "<|special_773|>",
-    "<|special_774|>",
-    "<|special_775|>",
-    "<|special_776|>",
-    "<|special_777|>",
-    "<|special_778|>",
-    "<|special_779|>",
-    "<|special_780|>",
-    "<|special_781|>",
-    "<|special_782|>",
-    "<|special_783|>",
-    "<|special_784|>",
-    "<|special_785|>",
-    "<|special_786|>",
-    "<|special_787|>",
-    "<|special_788|>",
-    "<|special_789|>",
-    "<|special_790|>",
-    "<|special_791|>",
-    "<|special_792|>",
-    "<|special_793|>",
-    "<|special_794|>",
-    "<|special_795|>",
-    "<|special_796|>",
-    "<|special_797|>",
-    "<|special_798|>",
-    "<|special_799|>",
-    "<|special_800|>",
-    "<|special_801|>",
-    "<|special_802|>",
-    "<|special_803|>",
-    "<|special_804|>",
-    "<|special_805|>",
-    "<|special_806|>",
-    "<|special_807|>",
-    "<|special_808|>",
-    "<|special_809|>",
-    "<|special_810|>",
-    "<|special_811|>",
-    "<|special_812|>",
-    "<|special_813|>",
-    "<|special_814|>",
-    "<|special_815|>",
-    "<|special_816|>",
-    "<|special_817|>",
-    "<|special_818|>",
-    "<|special_819|>",
-    "<|special_820|>",
-    "<|special_821|>",
-    "<|special_822|>",
-    "<|special_823|>",
-    "<|special_824|>",
-    "<|special_825|>",
-    "<|special_826|>",
-    "<|special_827|>",
-    "<|special_828|>",
-    "<|special_829|>",
-    "<|special_830|>",
-    "<|special_831|>",
-    "<|special_832|>",
-    "<|special_833|>",
-    "<|special_834|>",
-    "<|special_835|>",
-    "<|special_836|>",
-    "<|special_837|>",
-    "<|special_838|>",
-    "<|special_839|>",
-    "<|special_840|>",
-    "<|special_841|>",
-    "<|special_842|>",
-    "<|special_843|>",
-    "<|special_844|>",
-    "<|special_845|>",
-    "<|special_846|>",
-    "<|special_847|>",
-    "<|special_848|>",
-    "<|special_849|>",
-    "<|special_850|>",
-    "<|special_851|>",
-    "<|special_852|>",
-    "<|special_853|>",
-    "<|special_854|>",
-    "<|special_855|>",
-    "<|special_856|>",
-    "<|special_857|>",
-    "<|special_858|>",
-    "<|special_859|>",
-    "<|special_860|>",
-    "<|special_861|>",
-    "<|special_862|>",
-    "<|special_863|>",
-    "<|special_864|>",
-    "<|special_865|>",
-    "<|special_866|>",
-    "<|special_867|>",
-    "<|special_868|>",
-    "<|special_869|>",
-    "<|special_870|>",
-    "<|special_871|>",
-    "<|special_872|>",
-    "<|special_873|>",
-    "<|special_874|>",
-    "<|special_875|>",
-    "<|special_876|>",
-    "<|special_877|>",
-    "<|special_878|>",
-    "<|special_879|>",
-    "<|special_880|>",
-    "<|special_881|>",
-    "<|special_882|>",
-    "<|special_883|>",
-    "<|special_884|>",
-    "<|special_885|>",
-    "<|special_886|>",
-    "<|special_887|>",
-    "<|special_888|>",
-    "<|special_889|>",
-    "<|special_890|>",
-    "<|special_891|>",
-    "<|special_892|>",
-    "<|special_893|>",
-    "<|special_894|>",
-    "<|special_895|>",
-    "<|special_896|>",
-    "<|special_897|>",
-    "<|special_898|>",
-    "<|special_899|>",
-    "<|special_900|>",
-    "<|special_901|>",
-    "<|special_902|>",
-    "<|special_903|>",
-    "<|special_904|>",
-    "<|special_905|>",
-    "<|special_906|>",
-    "<|special_907|>",
-    "<|special_908|>",
-    "<|special_909|>",
-    "<|special_910|>",
-    "<|special_911|>",
-    "<|special_912|>",
-    "<|special_913|>",
-    "<|special_914|>",
-    "<|special_915|>",
-    "<|special_916|>",
-    "<|special_917|>",
-    "<|special_918|>",
-    "<|special_919|>",
-    "<|special_920|>",
-    "<|special_921|>",
-    "<|special_922|>",
-    "<|special_923|>",
-    "<|special_924|>",
-    "<|special_925|>",
-    "<|special_926|>",
-    "<|special_927|>",
-    "<|special_928|>",
-    "<|special_929|>",
-    "<|special_930|>",
-    "<|special_931|>",
-    "<|special_932|>",
-    "<|special_933|>",
-    "<|special_934|>",
-    "<|special_935|>",
-    "<|special_936|>",
-    "<|special_937|>",
-    "<|special_938|>",
-    "<|special_939|>",
-    "<|special_940|>",
-    "<|special_941|>",
-    "<|special_942|>",
-    "<|special_943|>",
-    "<|special_944|>",
-    "<|special_945|>",
-    "<|special_946|>",
-    "<|special_947|>",
-    "<|special_948|>",
-    "<|special_949|>",
-    "<|special_950|>",
-    "<|special_951|>",
-    "<|special_952|>",
-    "<|special_953|>",
-    "<|special_954|>",
-    "<|special_955|>",
-    "<|special_956|>",
-    "<|special_957|>",
-    "<|special_958|>",
-    "<|special_959|>",
-    "<|special_960|>",
-    "<|special_961|>",
-    "<|special_962|>",
-    "<|special_963|>",
-    "<|special_964|>",
-    "<|special_965|>",
-    "<|special_966|>",
-    "<|special_967|>",
-    "<|special_968|>",
-    "<|special_969|>",
-    "<|special_970|>",
-    "<|special_971|>",
-    "<|special_972|>",
-    "<|special_973|>",
-    "<|special_974|>",
-    "<|special_975|>",
-    "<|special_976|>",
-    "<|special_977|>",
-    "<|special_978|>",
-    "<|special_979|>",
-    "<|special_980|>",
-    "<|special_981|>",
-    "<|special_982|>",
-    "<|special_983|>",
-    "<|special_984|>",
-    "<|special_985|>",
-    "<|special_986|>",
-    "<|special_987|>",
-    "<|special_988|>",
-    "<|special_989|>",
-    "<|special_990|>",
-    "<|special_991|>",
-    "<|special_992|>",
-    "<|special_993|>",
-    "<|special_994|>",
-    "<|special_995|>",
-    "<|special_996|>",
-    "<|special_997|>",
-    "<|special_998|>",
-    "<|special_999|>",
-    "<|special_1000|>",
-    "<|special_1001|>",
-    "<|special_1002|>",
-    "<|special_1003|>",
-    "<|special_1004|>",
-    "<|special_1005|>",
-    "<|special_1006|>",
-    "<|special_1007|>",
-    "<|special_1008|>",
-    "<|special_1009|>",
-    "<|special_1010|>",
-    "<|special_1011|>",
-    "<|special_1012|>",
-    "<|special_1013|>",
-    "<|special_1014|>",
-    "<|special_1015|>",
-    "<|special_1016|>",
-    "<|special_1017|>",
-    "<|special_1018|>",
-    "<|special_1019|>",
-    "<|special_1020|>",
-    "<|special_1021|>",
-    "<|special_1022|>",
-    "<|special_1023|>",
-    "<|special_1024|>",
-    "<|special_1025|>",
-    "<|special_1026|>",
-    "<|special_1027|>",
-    "<|special_1028|>",
-    "<|special_1029|>",
-    "<|special_1030|>",
-    "<|special_1031|>",
-    "<|special_1032|>",
-    "<|special_1033|>",
-    "<|special_1034|>",
-    "<|special_1035|>",
-    "<|special_1036|>",
-    "<|special_1037|>",
-    "<|special_1038|>",
-    "<|special_1039|>",
-    "<|special_1040|>",
-    "<|special_1041|>",
-    "<|special_1042|>",
-    "<|special_1043|>",
-    "<|special_1044|>",
-    "<|special_1045|>",
-    "<|special_1046|>",
-    "<|special_1047|>",
-    "<|special_1048|>",
-    "<|special_1049|>",
-    "<|special_1050|>",
-    "<|special_1051|>",
-    "<|special_1052|>",
-    "<|special_1053|>",
-    "<|special_1054|>",
-    "<|special_1055|>",
-    "<|special_1056|>",
-    "<|special_1057|>",
-    "<|special_1058|>",
-    "<|special_1059|>",
-    "<|special_1060|>",
-    "<|special_1061|>",
-    "<|special_1062|>",
-    "<|special_1063|>",
-    "<|special_1064|>",
-    "<|special_1065|>",
-    "<|special_1066|>",
-    "<|special_1067|>",
-    "<|special_1068|>",
-    "<|special_1069|>",
-    "<|special_1070|>",
-    "<|special_1071|>",
-    "<|special_1072|>",
-    "<|special_1073|>",
-    "<|special_1074|>",
-    "<|special_1075|>",
-    "<|special_1076|>",
-    "<|special_1077|>",
-    "<|special_1078|>",
-    "<|special_1079|>",
-    "<|special_1080|>",
-    "<|special_1081|>",
-    "<|special_1082|>",
-    "<|special_1083|>",
-    "<|special_1084|>",
-    "<|special_1085|>",
-    "<|special_1086|>",
-    "<|special_1087|>",
-    "<|special_1088|>",
-    "<|special_1089|>",
-    "<|special_1090|>",
-    "<|special_1091|>",
-    "<|special_1092|>",
-    "<|special_1093|>",
-    "<|special_1094|>",
-    "<|special_1095|>",
-    "<|special_1096|>",
-    "<|special_1097|>",
-    "<|special_1098|>",
-    "<|special_1099|>",
-    "<|special_1100|>",
-    "<|special_1101|>",
-    "<|special_1102|>",
-    "<|special_1103|>",
-    "<|special_1104|>",
-    "<|special_1105|>",
-    "<|special_1106|>",
-    "<|special_1107|>",
-    "<|special_1108|>",
-    "<|special_1109|>",
-    "<|special_1110|>",
-    "<|special_1111|>",
-    "<|special_1112|>",
-    "<|special_1113|>",
-    "<|special_1114|>",
-    "<|special_1115|>",
-    "<|special_1116|>",
-    "<|special_1117|>",
-    "<|special_1118|>",
-    "<|special_1119|>",
-    "<|special_1120|>",
-    "<|special_1121|>",
-    "<|special_1122|>",
-    "<|special_1123|>",
-    "<|special_1124|>",
-    "<|special_1125|>",
-    "<|special_1126|>",
-    "<|special_1127|>",
-    "<|special_1128|>",
-    "<|special_1129|>",
-    "<|special_1130|>",
-    "<|special_1131|>",
-    "<|special_1132|>",
-    "<|special_1133|>",
-    "<|special_1134|>",
-    "<|special_1135|>",
-    "<|special_1136|>",
-    "<|special_1137|>",
-    "<|special_1138|>",
-    "<|special_1139|>",
-    "<|special_1140|>",
-    "<|special_1141|>",
-    "<|special_1142|>",
-    "<|special_1143|>",
-    "<|special_1144|>",
-    "<|special_1145|>",
-    "<|special_1146|>",
-    "<|special_1147|>",
-    "<|special_1148|>",
-    "<|special_1149|>",
-    "<|special_1150|>",
-    "<|special_1151|>",
-    "<|special_1152|>",
-    "<|special_1153|>",
-    "<|special_1154|>",
-    "<|special_1155|>",
-    "<|special_1156|>",
-    "<|special_1157|>",
-    "<|special_1158|>",
-    "<|special_1159|>",
-    "<|special_1160|>",
-    "<|special_1161|>",
-    "<|special_1162|>",
-    "<|special_1163|>",
-    "<|special_1164|>",
-    "<|special_1165|>",
-    "<|special_1166|>",
-    "<|special_1167|>",
-    "<|special_1168|>",
-    "<|special_1169|>",
-    "<|special_1170|>",
-    "<|special_1171|>",
-    "<|special_1172|>",
-    "<|special_1173|>",
-    "<|special_1174|>",
-    "<|special_1175|>",
-    "<|special_1176|>",
-    "<|special_1177|>",
-    "<|special_1178|>",
-    "<|special_1179|>",
-    "<|special_1180|>",
-    "<|special_1181|>",
-    "<|special_1182|>",
-    "<|special_1183|>",
-    "<|special_1184|>",
-    "<|special_1185|>",
-    "<|special_1186|>",
-    "<|special_1187|>",
-    "<|special_1188|>",
-    "<|special_1189|>",
-    "<|special_1190|>",
-    "<|special_1191|>",
-    "<|special_1192|>",
-    "<|special_1193|>",
-    "<|special_1194|>",
-    "<|special_1195|>",
-    "<|special_1196|>",
-    "<|special_1197|>",
-    "<|special_1198|>",
-    "<|special_1199|>",
-    "<|special_1200|>",
-    "<|special_1201|>",
-    "<|special_1202|>",
-    "<|special_1203|>",
-    "<|special_1204|>",
-    "<|special_1205|>",
-    "<|special_1206|>",
-    "<|special_1207|>",
-    "<|special_1208|>",
-    "<|special_1209|>",
-    "<|special_1210|>",
-    "<|special_1211|>",
-    "<|special_1212|>",
-    "<|special_1213|>",
-    "<|special_1214|>",
-    "<|special_1215|>",
-    "<|special_1216|>",
-    "<|special_1217|>",
-    "<|special_1218|>",
-    "<|special_1219|>",
-    "<|special_1220|>",
-    "<|special_1221|>",
-    "<|special_1222|>",
-    "<|special_1223|>",
-    "<|special_1224|>",
-    "<|special_1225|>",
-    "<|special_1226|>",
-    "<|special_1227|>",
-    "<|special_1228|>",
-    "<|special_1229|>",
-    "<|special_1230|>",
-    "<|special_1231|>",
-    "<|special_1232|>",
-    "<|special_1233|>",
-    "<|special_1234|>",
-    "<|special_1235|>",
-    "<|special_1236|>",
-    "<|special_1237|>",
-    "<|special_1238|>",
-    "<|special_1239|>",
-    "<|special_1240|>",
-    "<|special_1241|>",
-    "<|special_1242|>",
-    "<|special_1243|>",
-    "<|special_1244|>",
-    "<|special_1245|>",
-    "<|special_1246|>",
-    "<|special_1247|>",
-    "<|special_1248|>",
-    "<|special_1249|>",
-    "<|special_1250|>",
-    "<|special_1251|>",
-    "<|special_1252|>",
-    "<|special_1253|>",
-    "<|special_1254|>",
-    "<|special_1255|>",
-    "<|special_1256|>",
-    "<|special_1257|>",
-    "<|special_1258|>",
-    "<|special_1259|>",
-    "<|special_1260|>",
-    "<|special_1261|>",
-    "<|special_1262|>",
-    "<|special_1263|>",
-    "<|special_1264|>",
-    "<|special_1265|>",
-    "<|special_1266|>",
-    "<|special_1267|>",
-    "<|special_1268|>",
-    "<|special_1269|>",
-    "<|special_1270|>",
-    "<|special_1271|>",
-    "<|special_1272|>",
-    "<|special_1273|>",
-    "<|special_1274|>",
-    "<|special_1275|>",
-    "<|special_1276|>",
-    "<|special_1277|>",
-    "<|special_1278|>",
-    "<|special_1279|>",
-    "<|special_1280|>",
-    "<|special_1281|>",
-    "<|special_1282|>",
-    "<|special_1283|>",
-    "<|special_1284|>",
-    "<|special_1285|>",
-    "<|special_1286|>",
-    "<|special_1287|>",
-    "<|special_1288|>",
-    "<|special_1289|>",
-    "<|special_1290|>",
-    "<|special_1291|>",
-    "<|special_1292|>",
-    "<|special_1293|>",
-    "<|special_1294|>",
-    "<|special_1295|>",
-    "<|special_1296|>",
-    "<|special_1297|>",
-    "<|special_1298|>",
-    "<|special_1299|>",
-    "<|special_1300|>",
-    "<|special_1301|>",
-    "<|special_1302|>",
-    "<|special_1303|>",
-    "<|special_1304|>",
-    "<|special_1305|>",
-    "<|special_1306|>",
-    "<|special_1307|>",
-    "<|special_1308|>",
-    "<|special_1309|>",
-    "<|special_1310|>",
-    "<|special_1311|>",
-    "<|special_1312|>",
-    "<|special_1313|>",
-    "<|special_1314|>",
-    "<|special_1315|>",
-    "<|special_1316|>",
-    "<|special_1317|>",
-    "<|special_1318|>",
-    "<|special_1319|>",
-    "<|special_1320|>",
-    "<|special_1321|>",
-    "<|special_1322|>",
-    "<|special_1323|>",
-    "<|special_1324|>",
-    "<|special_1325|>",
-    "<|special_1326|>",
-    "<|special_1327|>",
-    "<|special_1328|>",
-    "<|special_1329|>",
-    "<|special_1330|>",
-    "<|special_1331|>",
-    "<|special_1332|>",
-    "<|special_1333|>",
-    "<|special_1334|>",
-    "<|special_1335|>",
-    "<|special_1336|>",
-    "<|special_1337|>",
-    "<|special_1338|>",
-    "<|special_1339|>",
-    "<|special_1340|>",
-    "<|special_1341|>",
-    "<|special_1342|>",
-    "<|special_1343|>",
-    "<|special_1344|>",
-    "<|special_1345|>",
-    "<|special_1346|>",
-    "<|special_1347|>",
-    "<|special_1348|>",
-    "<|special_1349|>",
-    "<|special_1350|>",
-    "<|special_1351|>",
-    "<|special_1352|>",
-    "<|special_1353|>",
-    "<|special_1354|>",
-    "<|special_1355|>",
-    "<|special_1356|>",
-    "<|special_1357|>",
-    "<|special_1358|>",
-    "<|special_1359|>",
-    "<|special_1360|>",
-    "<|special_1361|>",
-    "<|special_1362|>",
-    "<|special_1363|>",
-    "<|special_1364|>",
-    "<|special_1365|>",
-    "<|special_1366|>",
-    "<|special_1367|>",
-    "<|special_1368|>",
-    "<|special_1369|>",
-    "<|special_1370|>",
-    "<|special_1371|>",
-    "<|special_1372|>",
-    "<|special_1373|>",
-    "<|special_1374|>",
-    "<|special_1375|>",
-    "<|special_1376|>",
-    "<|special_1377|>",
-    "<|special_1378|>",
-    "<|special_1379|>",
-    "<|special_1380|>",
-    "<|special_1381|>",
-    "<|special_1382|>",
-    "<|special_1383|>",
-    "<|special_1384|>",
-    "<|special_1385|>",
-    "<|special_1386|>",
-    "<|special_1387|>",
-    "<|special_1388|>",
-    "<|special_1389|>",
-    "<|special_1390|>",
-    "<|special_1391|>",
-    "<|special_1392|>",
-    "<|special_1393|>",
-    "<|special_1394|>",
-    "<|special_1395|>",
-    "<|special_1396|>",
-    "<|special_1397|>",
-    "<|special_1398|>",
-    "<|special_1399|>",
-    "<|special_1400|>",
-    "<|special_1401|>",
-    "<|special_1402|>",
-    "<|special_1403|>",
-    "<|special_1404|>",
-    "<|special_1405|>",
-    "<|special_1406|>",
-    "<|special_1407|>",
-    "<|special_1408|>",
-    "<|special_1409|>",
-    "<|special_1410|>",
-    "<|special_1411|>",
-    "<|special_1412|>",
-    "<|special_1413|>",
-    "<|special_1414|>",
-    "<|special_1415|>",
-    "<|special_1416|>",
-    "<|special_1417|>",
-    "<|special_1418|>",
-    "<|special_1419|>",
-    "<|special_1420|>",
-    "<|special_1421|>",
-    "<|special_1422|>",
-    "<|special_1423|>",
-    "<|special_1424|>",
-    "<|special_1425|>",
-    "<|special_1426|>",
-    "<|special_1427|>",
-    "<|special_1428|>",
-    "<|special_1429|>",
-    "<|special_1430|>",
-    "<|special_1431|>",
-    "<|special_1432|>",
-    "<|special_1433|>",
-    "<|special_1434|>",
-    "<|special_1435|>",
-    "<|special_1436|>",
-    "<|special_1437|>",
-    "<|special_1438|>",
-    "<|special_1439|>",
-    "<|special_1440|>",
-    "<|special_1441|>",
-    "<|special_1442|>",
-    "<|special_1443|>",
-    "<|special_1444|>",
-    "<|special_1445|>",
-    "<|special_1446|>",
-    "<|special_1447|>",
-    "<|special_1448|>",
-    "<|special_1449|>",
-    "<|special_1450|>",
-    "<|special_1451|>",
-    "<|special_1452|>",
-    "<|special_1453|>",
-    "<|special_1454|>",
-    "<|special_1455|>",
-    "<|special_1456|>",
-    "<|special_1457|>",
-    "<|special_1458|>",
-    "<|special_1459|>",
-    "<|special_1460|>",
-    "<|special_1461|>",
-    "<|special_1462|>",
-    "<|special_1463|>",
-    "<|special_1464|>",
-    "<|special_1465|>",
-    "<|special_1466|>",
-    "<|special_1467|>",
-    "<|special_1468|>",
-    "<|special_1469|>",
-    "<|special_1470|>",
-    "<|special_1471|>",
-    "<|special_1472|>",
-    "<|special_1473|>",
-    "<|special_1474|>",
-    "<|special_1475|>",
-    "<|special_1476|>",
-    "<|special_1477|>",
-    "<|special_1478|>",
-    "<|special_1479|>",
-    "<|special_1480|>",
-    "<|special_1481|>",
-    "<|special_1482|>",
-    "<|special_1483|>",
-    "<|special_1484|>",
-    "<|special_1485|>",
-    "<|special_1486|>",
-    "<|special_1487|>",
-    "<|special_1488|>",
-    "<|special_1489|>",
-    "<|special_1490|>",
-    "<|special_1491|>",
-    "<|special_1492|>",
-    "<|special_1493|>",
-    "<|special_1494|>",
-    "<|special_1495|>",
-    "<|special_1496|>",
-    "<|special_1497|>",
-    "<|special_1498|>",
-    "<|special_1499|>",
-    "<|special_1500|>",
-    "<|special_1501|>",
-    "<|special_1502|>",
-    "<|special_1503|>",
-    "<|special_1504|>",
-    "<|special_1505|>",
-    "<|special_1506|>",
-    "<|special_1507|>",
-    "<|special_1508|>",
-    "<|special_1509|>",
-    "<|special_1510|>",
-    "<|special_1511|>",
-    "<|special_1512|>",
-    "<|special_1513|>",
-    "<|special_1514|>",
-    "<|special_1515|>",
-    "<|special_1516|>",
-    "<|special_1517|>",
-    "<|special_1518|>",
-    "<|special_1519|>",
-    "<|special_1520|>",
-    "<|special_1521|>",
-    "<|special_1522|>",
-    "<|special_1523|>",
-    "<|special_1524|>",
-    "<|special_1525|>",
-    "<|special_1526|>",
-    "<|special_1527|>",
-    "<|special_1528|>",
-    "<|special_1529|>",
-    "<|special_1530|>",
-    "<|special_1531|>",
-    "<|special_1532|>",
-    "<|special_1533|>",
-    "<|special_1534|>",
-    "<|special_1535|>",
-    "<|special_1536|>",
-    "<|special_1537|>",
-    "<|special_1538|>",
-    "<|special_1539|>",
-    "<|special_1540|>",
-    "<|special_1541|>",
-    "<|special_1542|>",
-    "<|special_1543|>",
-    "<|special_1544|>",
-    "<|special_1545|>",
-    "<|special_1546|>",
-    "<|special_1547|>",
-    "<|special_1548|>",
-    "<|special_1549|>",
-    "<|special_1550|>",
-    "<|special_1551|>",
-    "<|special_1552|>",
-    "<|special_1553|>",
-    "<|special_1554|>",
-    "<|special_1555|>",
-    "<|special_1556|>",
-    "<|special_1557|>",
-    "<|special_1558|>",
-    "<|special_1559|>",
-    "<|special_1560|>",
-    "<|special_1561|>",
-    "<|special_1562|>",
-    "<|special_1563|>",
-    "<|special_1564|>",
-    "<|special_1565|>",
-    "<|special_1566|>",
-    "<|special_1567|>",
-    "<|special_1568|>",
-    "<|special_1569|>",
-    "<|special_1570|>",
-    "<|special_1571|>",
-    "<|special_1572|>",
-    "<|special_1573|>",
-    "<|special_1574|>",
-    "<|special_1575|>",
-    "<|special_1576|>",
-    "<|special_1577|>",
-    "<|special_1578|>",
-    "<|special_1579|>",
-    "<|special_1580|>",
-    "<|special_1581|>",
-    "<|special_1582|>",
-    "<|special_1583|>",
-    "<|special_1584|>",
-    "<|special_1585|>",
-    "<|special_1586|>",
-    "<|special_1587|>",
-    "<|special_1588|>",
-    "<|special_1589|>",
-    "<|special_1590|>",
-    "<|special_1591|>",
-    "<|special_1592|>",
-    "<|special_1593|>",
-    "<|special_1594|>",
-    "<|special_1595|>",
-    "<|special_1596|>",
-    "<|special_1597|>",
-    "<|special_1598|>",
-    "<|special_1599|>",
-    "<|special_1600|>",
-    "<|special_1601|>",
-    "<|special_1602|>",
-    "<|special_1603|>",
-    "<|special_1604|>",
-    "<|special_1605|>",
-    "<|special_1606|>",
-    "<|special_1607|>",
-    "<|special_1608|>",
-    "<|special_1609|>",
-    "<|special_1610|>",
-    "<|special_1611|>",
-    "<|special_1612|>",
-    "<|special_1613|>",
-    "<|special_1614|>",
-    "<|special_1615|>",
-    "<|special_1616|>",
-    "<|special_1617|>",
-    "<|special_1618|>",
-    "<|special_1619|>",
-    "<|special_1620|>",
-    "<|special_1621|>",
-    "<|special_1622|>",
-    "<|special_1623|>",
-    "<|special_1624|>",
-    "<|special_1625|>",
-    "<|special_1626|>",
-    "<|special_1627|>",
-    "<|special_1628|>",
-    "<|special_1629|>",
-    "<|special_1630|>",
-    "<|special_1631|>",
-    "<|special_1632|>",
-    "<|special_1633|>",
-    "<|special_1634|>",
-    "<|special_1635|>",
-    "<|special_1636|>",
-    "<|special_1637|>",
-    "<|special_1638|>",
-    "<|special_1639|>",
-    "<|special_1640|>",
-    "<|special_1641|>",
-    "<|special_1642|>",
-    "<|special_1643|>",
-    "<|special_1644|>",
-    "<|special_1645|>",
-    "<|special_1646|>",
-    "<|special_1647|>",
-    "<|special_1648|>",
-    "<|special_1649|>",
-    "<|special_1650|>",
-    "<|special_1651|>",
-    "<|special_1652|>",
-    "<|special_1653|>",
-    "<|special_1654|>",
-    "<|special_1655|>",
-    "<|special_1656|>",
-    "<|special_1657|>",
-    "<|special_1658|>",
-    "<|special_1659|>",
-    "<|special_1660|>",
-    "<|special_1661|>",
-    "<|special_1662|>",
-    "<|special_1663|>",
-    "<|special_1664|>",
-    "<|special_1665|>",
-    "<|special_1666|>",
-    "<|special_1667|>",
-    "<|special_1668|>",
-    "<|special_1669|>",
-    "<|special_1670|>",
-    "<|special_1671|>",
-    "<|special_1672|>",
-    "<|special_1673|>",
-    "<|special_1674|>",
-    "<|special_1675|>",
-    "<|special_1676|>",
-    "<|special_1677|>",
-    "<|special_1678|>",
-    "<|special_1679|>",
-    "<|special_1680|>",
-    "<|special_1681|>",
-    "<|special_1682|>",
-    "<|special_1683|>",
-    "<|special_1684|>",
-    "<|special_1685|>",
-    "<|special_1686|>",
-    "<|special_1687|>",
-    "<|special_1688|>",
-    "<|special_1689|>",
-    "<|special_1690|>",
-    "<|special_1691|>",
-    "<|special_1692|>",
-    "<|special_1693|>",
-    "<|special_1694|>",
-    "<|special_1695|>",
-    "<|special_1696|>",
-    "<|special_1697|>",
-    "<|special_1698|>",
-    "<|special_1699|>",
-    "<|special_1700|>",
-    "<|special_1701|>",
-    "<|special_1702|>",
-    "<|special_1703|>",
-    "<|special_1704|>",
-    "<|special_1705|>",
-    "<|special_1706|>",
-    "<|special_1707|>",
-    "<|special_1708|>",
-    "<|special_1709|>",
-    "<|special_1710|>",
-    "<|special_1711|>",
-    "<|special_1712|>",
-    "<|special_1713|>",
-    "<|special_1714|>",
-    "<|special_1715|>",
-    "<|special_1716|>",
-    "<|special_1717|>",
-    "<|special_1718|>",
-    "<|special_1719|>",
-    "<|special_1720|>",
-    "<|special_1721|>",
-    "<|special_1722|>",
-    "<|special_1723|>",
-    "<|special_1724|>",
-    "<|special_1725|>",
-    "<|special_1726|>",
-    "<|special_1727|>",
-    "<|special_1728|>",
-    "<|special_1729|>",
-    "<|special_1730|>",
-    "<|special_1731|>",
-    "<|special_1732|>",
-    "<|special_1733|>",
-    "<|special_1734|>",
-    "<|special_1735|>",
-    "<|special_1736|>",
-    "<|special_1737|>",
-    "<|special_1738|>",
-    "<|special_1739|>",
-    "<|special_1740|>",
-    "<|special_1741|>",
-    "<|special_1742|>",
-    "<|special_1743|>",
-    "<|special_1744|>",
-    "<|special_1745|>",
-    "<|special_1746|>",
-    "<|special_1747|>",
-    "<|special_1748|>",
-    "<|special_1749|>",
-    "<|special_1750|>",
-    "<|special_1751|>",
-    "<|special_1752|>",
-    "<|special_1753|>",
-    "<|special_1754|>",
-    "<|special_1755|>",
-    "<|special_1756|>",
-    "<|special_1757|>",
-    "<|special_1758|>",
-    "<|special_1759|>",
-    "<|special_1760|>",
-    "<|special_1761|>",
-    "<|special_1762|>",
-    "<|special_1763|>",
-    "<|special_1764|>",
-    "<|special_1765|>",
-    "<|special_1766|>",
-    "<|special_1767|>",
-    "<|special_1768|>",
-    "<|special_1769|>",
-    "<|special_1770|>",
-    "<|special_1771|>",
-    "<|special_1772|>",
-    "<|special_1773|>",
-    "<|special_1774|>",
-    "<|special_1775|>",
-    "<|special_1776|>",
-    "<|special_1777|>",
-    "<|special_1778|>",
-    "<|special_1779|>",
-    "<|special_1780|>",
-    "<|special_1781|>",
-    "<|special_1782|>",
-    "<|special_1783|>",
-    "<|special_1784|>",
-    "<|special_1785|>",
-    "<|special_1786|>",
-    "<|special_1787|>",
-    "<|special_1788|>",
-    "<|special_1789|>",
-    "<|special_1790|>",
-    "<|special_1791|>",
-    "<|special_1792|>",
-    "<|special_1793|>",
-    "<|special_1794|>",
-    "<|special_1795|>",
-    "<|special_1796|>",
-    "<|special_1797|>",
-    "<|special_1798|>",
-    "<|special_1799|>",
-    "<|special_1800|>",
-    "<|special_1801|>",
-    "<|special_1802|>",
-    "<|special_1803|>",
-    "<|special_1804|>",
-    "<|special_1805|>",
-    "<|special_1806|>",
-    "<|special_1807|>",
-    "<|special_1808|>",
-    "<|special_1809|>",
-    "<|special_1810|>",
-    "<|special_1811|>",
-    "<|special_1812|>",
-    "<|special_1813|>",
-    "<|special_1814|>",
-    "<|special_1815|>",
-    "<|special_1816|>",
-    "<|special_1817|>",
-    "<|special_1818|>",
-    "<|special_1819|>",
-    "<|special_1820|>",
-    "<|special_1821|>",
-    "<|special_1822|>",
-    "<|special_1823|>",
-    "<|special_1824|>",
-    "<|special_1825|>",
-    "<|special_1826|>",
-    "<|special_1827|>",
-    "<|special_1828|>",
-    "<|special_1829|>",
-    "<|special_1830|>",
-    "<|special_1831|>",
-    "<|special_1832|>",
-    "<|special_1833|>",
-    "<|special_1834|>",
-    "<|special_1835|>",
-    "<|special_1836|>",
-    "<|special_1837|>",
-    "<|special_1838|>",
-    "<|special_1839|>",
-    "<|special_1840|>",
-    "<|special_1841|>",
-    "<|special_1842|>",
-    "<|special_1843|>",
-    "<|special_1844|>",
-    "<|special_1845|>",
-    "<|special_1846|>",
-    "<|special_1847|>",
-    "<|special_1848|>",
-    "<|special_1849|>",
-    "<|special_1850|>",
-    "<|special_1851|>",
-    "<|special_1852|>",
-    "<|special_1853|>",
-    "<|special_1854|>",
-    "<|special_1855|>",
-    "<|special_1856|>",
-    "<|special_1857|>",
-    "<|special_1858|>",
-    "<|special_1859|>",
-    "<|special_1860|>",
-    "<|special_1861|>",
-    "<|special_1862|>",
-    "<|special_1863|>",
-    "<|special_1864|>",
-    "<|special_1865|>",
-    "<|special_1866|>",
-    "<|special_1867|>",
-    "<|special_1868|>",
-    "<|special_1869|>",
-    "<|special_1870|>",
-    "<|special_1871|>",
-    "<|special_1872|>",
-    "<|special_1873|>",
-    "<|special_1874|>",
-    "<|special_1875|>",
-    "<|special_1876|>",
-    "<|special_1877|>",
-    "<|special_1878|>",
-    "<|special_1879|>",
-    "<|special_1880|>",
-    "<|special_1881|>",
-    "<|special_1882|>",
-    "<|special_1883|>",
-    "<|special_1884|>",
-    "<|special_1885|>",
-    "<|special_1886|>",
-    "<|special_1887|>",
-    "<|special_1888|>",
-    "<|special_1889|>",
-    "<|special_1890|>",
-    "<|special_1891|>",
-    "<|special_1892|>",
-    "<|special_1893|>",
-    "<|special_1894|>",
-    "<|special_1895|>",
-    "<|special_1896|>",
-    "<|special_1897|>",
-    "<|special_1898|>",
-    "<|special_1899|>",
-    "<|special_1900|>",
-    "<|special_1901|>",
-    "<|special_1902|>",
-    "<|special_1903|>",
-    "<|special_1904|>",
-    "<|special_1905|>",
-    "<|special_1906|>",
-    "<|special_1907|>",
-    "<|special_1908|>",
-    "<|special_1909|>",
-    "<|special_1910|>",
-    "<|special_1911|>",
-    "<|special_1912|>",
-    "<|special_1913|>",
-    "<|special_1914|>",
-    "<|special_1915|>",
-    "<|special_1916|>",
-    "<|special_1917|>",
-    "<|special_1918|>",
-    "<|special_1919|>",
-    "<|special_1920|>",
-    "<|special_1921|>",
-    "<|special_1922|>",
-    "<|special_1923|>",
-    "<|special_1924|>",
-    "<|special_1925|>",
-    "<|special_1926|>",
-    "<|special_1927|>",
-    "<|special_1928|>",
-    "<|special_1929|>",
-    "<|special_1930|>",
-    "<|special_1931|>",
-    "<|special_1932|>",
-    "<|special_1933|>",
-    "<|special_1934|>",
-    "<|special_1935|>",
-    "<|special_1936|>",
-    "<|special_1937|>",
-    "<|special_1938|>",
-    "<|special_1939|>",
-    "<|special_1940|>",
-    "<|special_1941|>",
-    "<|special_1942|>",
-    "<|special_1943|>",
-    "<|special_1944|>",
-    "<|special_1945|>",
-    "<|special_1946|>",
-    "<|special_1947|>",
-    "<|special_1948|>",
-    "<|special_1949|>",
-    "<|special_1950|>",
-    "<|special_1951|>",
-    "<|special_1952|>",
-    "<|special_1953|>",
-    "<|special_1954|>",
-    "<|special_1955|>",
-    "<|special_1956|>",
-    "<|special_1957|>",
-    "<|special_1958|>",
-    "<|special_1959|>",
-    "<|special_1960|>",
-    "<|special_1961|>",
-    "<|special_1962|>",
-    "<|special_1963|>",
-    "<|special_1964|>",
-    "<|special_1965|>",
-    "<|special_1966|>",
-    "<|special_1967|>",
-    "<|special_1968|>",
-    "<|special_1969|>",
-    "<|special_1970|>",
-    "<|special_1971|>",
-    "<|special_1972|>",
-    "<|special_1973|>",
-    "<|special_1974|>",
-    "<|special_1975|>",
-    "<|special_1976|>",
-    "<|special_1977|>",
-    "<|special_1978|>",
-    "<|special_1979|>",
-    "<|special_1980|>",
-    "<|special_1981|>",
-    "<|special_1982|>",
-    "<|special_1983|>",
-    "<|special_1984|>",
-    "<|special_1985|>",
-    "<|special_1986|>",
-    "<|special_1987|>",
-    "<|special_1988|>",
-    "<|special_1989|>",
-    "<|special_1990|>",
-    "<|special_1991|>",
-    "<|special_1992|>",
-    "<|special_1993|>",
-    "<|special_1994|>",
-    "<|special_1995|>",
-    "<|special_1996|>",
-    "<|special_1997|>",
-    "<|special_1998|>",
-    "<|special_1999|>",
-    "<|special_2000|>",
-    "<|special_2001|>",
-    "<|special_2002|>",
-    "<|special_2003|>",
-    "<|special_2004|>",
-    "<|special_2005|>",
-    "<|special_2006|>",
-    "<|special_2007|>",
-    "<|special_2008|>",
-    "<|special_2009|>",
-    "<|special_2010|>",
-    "<|special_2011|>",
-    "<|special_2012|>",
-    "<|special_2013|>",
-    "<|special_2014|>",
-    "<|special_2015|>",
-    "<|special_2016|>",
-    "<|special_2017|>",
-    "<|special_2018|>",
-    "<|special_2019|>",
-    "<|special_2020|>",
-    "<|special_2021|>",
-    "<|special_2022|>",
-    "<|special_2023|>",
-    "<|special_2024|>",
-    "<|special_2025|>",
-    "<|special_2026|>",
-    "<|special_2027|>",
-    "<|special_2028|>",
-    "<|special_2029|>",
-    "<|special_2030|>",
-    "<|special_2031|>",
-    "<|special_2032|>",
-    "<|special_2033|>",
-    "<|special_2034|>",
-    "<|special_2035|>",
-    "<|special_2036|>",
-    "<|special_2037|>",
-    "<|special_2038|>",
-    "<|special_2039|>",
-    "<|special_2040|>",
-    "<|special_2041|>",
-    "<|special_2042|>",
-    "<|special_2043|>",
-    "<|special_2044|>",
-    "<|special_2045|>",
-    "<|special_2046|>",
-    "<|special_2047|>",
-    "<|special_2048|>",
-    "<|special_2049|>",
-    "<|special_2050|>",
-    "<|special_2051|>",
-    "<|special_2052|>",
-    "<|special_2053|>",
-    "<|special_2054|>",
-    "<|special_2055|>",
-    "<|special_2056|>",
-    "<|special_2057|>",
-    "<|special_2058|>",
-    "<|special_2059|>",
-    "<|special_2060|>",
-    "<|special_2061|>",
-    "<|special_2062|>",
-    "<|special_2063|>",
-    "<|special_2064|>",
-    "<|special_2065|>",
-    "<|special_2066|>",
-    "<|special_2067|>",
-    "<|special_2068|>",
-    "<|special_2069|>",
-    "<|special_2070|>",
-    "<|special_2071|>",
-    "<|special_2072|>",
-    "<|special_2073|>",
-    "<|special_2074|>",
-    "<|special_2075|>",
-    "<|special_2076|>",
-    "<|special_2077|>",
-    "<|special_2078|>",
-    "<|special_2079|>",
-    "<|special_2080|>",
-    "<|special_2081|>",
-    "<|special_2082|>",
-    "<|special_2083|>",
-    "<|special_2084|>",
-    "<|special_2085|>",
-    "<|special_2086|>",
-    "<|special_2087|>",
-    "<|special_2088|>",
-    "<|special_2089|>",
-    "<|special_2090|>",
-    "<|special_2091|>",
-    "<|special_2092|>",
-    "<|special_2093|>",
-    "<|special_2094|>",
-    "<|special_2095|>",
-    "<|special_2096|>",
-    "<|special_2097|>",
-    "<|special_2098|>",
-    "<|special_2099|>",
-    "<|special_2100|>",
-    "<|special_2101|>",
-    "<|special_2102|>",
-    "<|special_2103|>",
-    "<|special_2104|>",
-    "<|special_2105|>",
-    "<|special_2106|>",
-    "<|special_2107|>",
-    "<|special_2108|>",
-    "<|special_2109|>",
-    "<|special_2110|>",
-    "<|special_2111|>",
-    "<|special_2112|>",
-    "<|special_2113|>",
-    "<|special_2114|>",
-    "<|special_2115|>",
-    "<|special_2116|>",
-    "<|special_2117|>",
-    "<|special_2118|>",
-    "<|special_2119|>",
-    "<|special_2120|>",
-    "<|special_2121|>",
-    "<|special_2122|>",
-    "<|special_2123|>",
-    "<|special_2124|>",
-    "<|special_2125|>",
-    "<|special_2126|>",
-    "<|special_2127|>",
-    "<|special_2128|>",
-    "<|special_2129|>",
-    "<|special_2130|>",
-    "<|special_2131|>",
-    "<|special_2132|>",
-    "<|special_2133|>",
-    "<|special_2134|>",
-    "<|special_2135|>",
-    "<|special_2136|>",
-    "<|special_2137|>",
-    "<|special_2138|>",
-    "<|special_2139|>",
-    "<|special_2140|>",
-    "<|special_2141|>",
-    "<|special_2142|>",
-    "<|special_2143|>",
-    "<|special_2144|>",
-    "<|special_2145|>",
-    "<|special_2146|>",
-    "<|special_2147|>",
-    "<|special_2148|>",
-    "<|special_2149|>",
-    "<|special_2150|>",
-    "<|special_2151|>",
-    "<|special_2152|>",
-    "<|special_2153|>",
-    "<|special_2154|>",
-    "<|special_2155|>",
-    "<|special_2156|>",
-    "<|special_2157|>",
-    "<|special_2158|>",
-    "<|special_2159|>",
-    "<|special_2160|>",
-    "<|special_2161|>",
-    "<|special_2162|>",
-    "<|special_2163|>",
-    "<|special_2164|>",
-    "<|special_2165|>",
-    "<|special_2166|>",
-    "<|special_2167|>",
-    "<|special_2168|>",
-    "<|special_2169|>",
-    "<|special_2170|>",
-    "<|special_2171|>",
-    "<|special_2172|>",
-    "<|special_2173|>",
-    "<|special_2174|>",
-    "<|special_2175|>",
-    "<|special_2176|>",
-    "<|special_2177|>",
-    "<|special_2178|>",
-    "<|special_2179|>",
-    "<|special_2180|>",
-    "<|special_2181|>",
-    "<|special_2182|>",
-    "<|special_2183|>",
-    "<|special_2184|>",
-    "<|special_2185|>",
-    "<|special_2186|>",
-    "<|special_2187|>",
-    "<|special_2188|>",
-    "<|special_2189|>",
-    "<|special_2190|>",
-    "<|special_2191|>",
-    "<|special_2192|>",
-    "<|special_2193|>",
-    "<|special_2194|>",
-    "<|special_2195|>",
-    "<|special_2196|>",
-    "<|special_2197|>",
-    "<|special_2198|>",
-    "<|special_2199|>",
-    "<|special_2200|>",
-    "<|special_2201|>",
-    "<|special_2202|>",
-    "<|special_2203|>",
-    "<|special_2204|>",
-    "<|special_2205|>",
-    "<|special_2206|>",
-    "<|special_2207|>",
-    "<|special_2208|>",
-    "<|special_2209|>",
-    "<|special_2210|>",
-    "<|special_2211|>",
-    "<|special_2212|>",
-    "<|special_2213|>",
-    "<|special_2214|>",
-    "<|special_2215|>",
-    "<|special_2216|>",
-    "<|special_2217|>",
-    "<|special_2218|>",
-    "<|special_2219|>",
-    "<|special_2220|>",
-    "<|special_2221|>",
-    "<|special_2222|>",
-    "<|special_2223|>",
-    "<|special_2224|>",
-    "<|special_2225|>",
-    "<|special_2226|>",
-    "<|special_2227|>",
-    "<|special_2228|>",
-    "<|special_2229|>",
-    "<|special_2230|>",
-    "<|special_2231|>",
-    "<|special_2232|>",
-    "<|special_2233|>",
-    "<|special_2234|>",
-    "<|special_2235|>",
-    "<|special_2236|>",
-    "<|special_2237|>",
-    "<|special_2238|>",
-    "<|special_2239|>",
-    "<|special_2240|>",
-    "<|special_2241|>",
-    "<|special_2242|>",
-    "<|special_2243|>",
-    "<|special_2244|>",
-    "<|special_2245|>",
-    "<|special_2246|>",
-    "<|special_2247|>",
-    "<|special_2248|>",
-    "<|special_2249|>",
-    "<|special_2250|>",
-    "<|special_2251|>",
-    "<|special_2252|>",
-    "<|special_2253|>",
-    "<|special_2254|>",
-    "<|special_2255|>",
-    "<|special_2256|>",
-    "<|special_2257|>",
-    "<|special_2258|>",
-    "<|special_2259|>",
-    "<|special_2260|>",
-    "<|special_2261|>",
-    "<|special_2262|>",
-    "<|special_2263|>",
-    "<|special_2264|>",
-    "<|special_2265|>",
-    "<|special_2266|>",
-    "<|special_2267|>",
-    "<|special_2268|>",
-    "<|special_2269|>",
-    "<|special_2270|>",
-    "<|special_2271|>",
-    "<|special_2272|>",
-    "<|special_2273|>",
-    "<|special_2274|>",
-    "<|special_2275|>",
-    "<|special_2276|>",
-    "<|special_2277|>",
-    "<|special_2278|>",
-    "<|special_2279|>",
-    "<|special_2280|>",
-    "<|special_2281|>",
-    "<|special_2282|>",
-    "<|special_2283|>",
-    "<|special_2284|>",
-    "<|special_2285|>",
-    "<|special_2286|>",
-    "<|special_2287|>",
-    "<|special_2288|>",
-    "<|special_2289|>",
-    "<|special_2290|>",
-    "<|special_2291|>",
-    "<|special_2292|>",
-    "<|special_2293|>",
-    "<|special_2294|>",
-    "<|special_2295|>",
-    "<|special_2296|>",
-    "<|special_2297|>",
-    "<|special_2298|>",
-    "<|special_2299|>",
-    "<|special_2300|>",
-    "<|special_2301|>",
-    "<|special_2302|>",
-    "<|special_2303|>",
-    "<|special_2304|>",
-    "<|special_2305|>",
-    "<|special_2306|>",
-    "<|special_2307|>",
-    "<|special_2308|>",
-    "<|special_2309|>",
-    "<|special_2310|>",
-    "<|special_2311|>",
-    "<|special_2312|>",
-    "<|special_2313|>",
-    "<|special_2314|>",
-    "<|special_2315|>",
-    "<|special_2316|>",
-    "<|special_2317|>",
-    "<|special_2318|>",
-    "<|special_2319|>",
-    "<|special_2320|>",
-    "<|special_2321|>",
-    "<|special_2322|>",
-    "<|special_2323|>",
-    "<|special_2324|>",
-    "<|special_2325|>",
-    "<|special_2326|>",
-    "<|special_2327|>",
-    "<|special_2328|>",
-    "<|special_2329|>",
-    "<|special_2330|>",
-    "<|special_2331|>",
-    "<|special_2332|>",
-    "<|special_2333|>",
-    "<|special_2334|>",
-    "<|special_2335|>",
-    "<|special_2336|>",
-    "<|special_2337|>",
-    "<|special_2338|>",
-    "<|special_2339|>",
-    "<|special_2340|>",
-    "<|special_2341|>",
-    "<|special_2342|>",
-    "<|special_2343|>",
-    "<|special_2344|>",
-    "<|special_2345|>",
-    "<|special_2346|>",
-    "<|special_2347|>",
-    "<|special_2348|>",
-    "<|special_2349|>",
-    "<|special_2350|>",
-    "<|special_2351|>",
-    "<|special_2352|>",
-    "<|special_2353|>",
-    "<|special_2354|>",
-    "<|special_2355|>",
-    "<|special_2356|>",
-    "<|special_2357|>",
-    "<|special_2358|>",
-    "<|special_2359|>",
-    "<|special_2360|>",
-    "<|special_2361|>",
-    "<|special_2362|>",
-    "<|special_2363|>",
-    "<|special_2364|>",
-    "<|special_2365|>",
-    "<|special_2366|>",
-    "<|special_2367|>",
-    "<|special_2368|>",
-    "<|special_2369|>",
-    "<|special_2370|>",
-    "<|special_2371|>",
-    "<|special_2372|>",
-    "<|special_2373|>",
-    "<|special_2374|>",
-    "<|special_2375|>",
-    "<|special_2376|>",
-    "<|special_2377|>",
-    "<|special_2378|>",
-    "<|special_2379|>",
-    "<|special_2380|>",
-    "<|special_2381|>",
-    "<|special_2382|>",
-    "<|special_2383|>",
-    "<|special_2384|>",
-    "<|special_2385|>",
-    "<|special_2386|>",
-    "<|special_2387|>",
-    "<|special_2388|>",
-    "<|special_2389|>",
-    "<|special_2390|>",
-    "<|special_2391|>",
-    "<|special_2392|>",
-    "<|special_2393|>",
-    "<|special_2394|>",
-    "<|special_2395|>",
-    "<|special_2396|>",
-    "<|special_2397|>",
-    "<|special_2398|>",
-    "<|special_2399|>",
-    "<|special_2400|>",
-    "<|special_2401|>",
-    "<|special_2402|>",
-    "<|special_2403|>",
-    "<|special_2404|>",
-    "<|special_2405|>",
-    "<|special_2406|>",
-    "<|special_2407|>",
-    "<|special_2408|>",
-    "<|special_2409|>",
-    "<|special_2410|>",
-    "<|special_2411|>",
-    "<|special_2412|>",
-    "<|special_2413|>",
-    "<|special_2414|>",
-    "<|special_2415|>",
-    "<|special_2416|>",
-    "<|special_2417|>",
-    "<|special_2418|>",
-    "<|special_2419|>",
-    "<|special_2420|>",
-    "<|special_2421|>",
-    "<|special_2422|>",
-    "<|special_2423|>",
-    "<|special_2424|>",
-    "<|special_2425|>",
-    "<|special_2426|>",
-    "<|special_2427|>",
-    "<|special_2428|>",
-    "<|special_2429|>",
-    "<|special_2430|>",
-    "<|special_2431|>",
-    "<|special_2432|>",
-    "<|special_2433|>",
-    "<|special_2434|>",
-    "<|special_2435|>",
-    "<|special_2436|>",
-    "<|special_2437|>",
-    "<|special_2438|>",
-    "<|special_2439|>",
-    "<|special_2440|>",
-    "<|special_2441|>",
-    "<|special_2442|>",
-    "<|special_2443|>",
-    "<|special_2444|>",
-    "<|special_2445|>",
-    "<|special_2446|>",
-    "<|special_2447|>",
-    "<|special_2448|>",
-    "<|special_2449|>",
-    "<|special_2450|>",
-    "<|special_2451|>",
-    "<|special_2452|>",
-    "<|special_2453|>",
-    "<|special_2454|>",
-    "<|special_2455|>",
-    "<|special_2456|>",
-    "<|special_2457|>",
-    "<|special_2458|>",
-    "<|special_2459|>",
-    "<|special_2460|>",
-    "<|special_2461|>",
-    "<|special_2462|>",
-    "<|special_2463|>",
-    "<|special_2464|>",
-    "<|special_2465|>",
-    "<|special_2466|>",
-    "<|special_2467|>",
-    "<|special_2468|>",
-    "<|special_2469|>",
-    "<|special_2470|>",
-    "<|special_2471|>",
-    "<|special_2472|>",
-    "<|special_2473|>",
-    "<|special_2474|>",
-    "<|special_2475|>",
-    "<|special_2476|>",
-    "<|special_2477|>",
-    "<|special_2478|>",
-    "<|special_2479|>",
-    "<|special_2480|>",
-    "<|special_2481|>",
-    "<|special_2482|>",
-    "<|special_2483|>",
-    "<|special_2484|>",
-    "<|special_2485|>",
-    "<|special_2486|>",
-    "<|special_2487|>",
-    "<|special_2488|>",
-    "<|special_2489|>",
-    "<|special_2490|>",
-    "<|special_2491|>",
-    "<|special_2492|>",
-    "<|special_2493|>",
-    "<|special_2494|>",
-    "<|special_2495|>",
-    "<|special_2496|>",
-    "<|special_2497|>",
-    "<|special_2498|>",
-    "<|special_2499|>",
-    "<|special_2500|>",
-    "<|special_2501|>",
-    "<|special_2502|>",
-    "<|special_2503|>",
-    "<|special_2504|>",
-    "<|special_2505|>",
-    "<|special_2506|>",
-    "<|special_2507|>",
-    "<|special_2508|>",
-    "<|special_2509|>",
-    "<|special_2510|>",
-    "<|special_2511|>",
-    "<|special_2512|>",
-    "<|special_2513|>",
-    "<|special_2514|>",
-    "<|special_2515|>",
-    "<|special_2516|>",
-    "<|special_2517|>",
-    "<|special_2518|>",
-    "<|special_2519|>",
-    "<|special_2520|>",
-    "<|special_2521|>",
-    "<|special_2522|>",
-    "<|special_2523|>",
-    "<|special_2524|>",
-    "<|special_2525|>",
-    "<|special_2526|>",
-    "<|special_2527|>",
-    "<|special_2528|>",
-    "<|special_2529|>",
-    "<|special_2530|>",
-    "<|special_2531|>",
-    "<|special_2532|>",
-    "<|special_2533|>",
-    "<|special_2534|>",
-    "<|special_2535|>",
-    "<|special_2536|>",
-    "<|special_2537|>",
-    "<|special_2538|>",
-    "<|special_2539|>",
-    "<|special_2540|>",
-    "<|special_2541|>",
-    "<|special_2542|>",
-    "<|special_2543|>",
-    "<|special_2544|>",
-    "<|special_2545|>",
-    "<|special_2546|>",
-    "<|special_2547|>",
-    "<|special_2548|>",
-    "<|special_2549|>",
-    "<|special_2550|>",
-    "<|special_2551|>",
-    "<|special_2552|>",
-    "<|special_2553|>",
-    "<|special_2554|>",
-    "<|special_2555|>",
-    "<|special_2556|>",
-    "<|special_2557|>",
-    "<|special_2558|>",
-    "<|special_2559|>",
-    "<|special_2560|>",
-    "<|special_2561|>",
-    "<|special_2562|>",
-    "<|special_2563|>",
-    "<|special_2564|>",
-    "<|special_2565|>",
-    "<|special_2566|>",
-    "<|special_2567|>",
-    "<|special_2568|>",
-    "<|special_2569|>",
-    "<|special_2570|>",
-    "<|special_2571|>",
-    "<|special_2572|>",
-    "<|special_2573|>",
-    "<|special_2574|>",
-    "<|special_2575|>",
-    "<|special_2576|>",
-    "<|special_2577|>",
-    "<|special_2578|>",
-    "<|special_2579|>",
-    "<|special_2580|>",
-    "<|special_2581|>",
-    "<|special_2582|>",
-    "<|special_2583|>",
-    "<|special_2584|>",
-    "<|special_2585|>",
-    "<|special_2586|>",
-    "<|special_2587|>",
-    "<|special_2588|>",
-    "<|special_2589|>",
-    "<|special_2590|>",
-    "<|special_2591|>",
-    "<|special_2592|>",
-    "<|special_2593|>",
-    "<|special_2594|>",
-    "<|special_2595|>",
-    "<|special_2596|>",
-    "<|special_2597|>",
-    "<|special_2598|>",
-    "<|special_2599|>",
-    "<|special_2600|>",
-    "<|special_2601|>",
-    "<|special_2602|>",
-    "<|special_2603|>",
-    "<|special_2604|>",
-    "<|special_2605|>",
-    "<|special_2606|>",
-    "<|special_2607|>",
-    "<|special_2608|>",
-    "<|special_2609|>",
-    "<|special_2610|>",
-    "<|special_2611|>",
-    "<|special_2612|>",
-    "<|special_2613|>",
-    "<|special_2614|>",
-    "<|special_2615|>",
-    "<|special_2616|>",
-    "<|special_2617|>",
-    "<|special_2618|>",
-    "<|special_2619|>",
-    "<|special_2620|>",
-    "<|special_2621|>",
-    "<|special_2622|>",
-    "<|special_2623|>",
-    "<|special_2624|>",
-    "<|special_2625|>",
-    "<|special_2626|>",
-    "<|special_2627|>",
-    "<|special_2628|>",
-    "<|special_2629|>",
-    "<|special_2630|>",
-    "<|special_2631|>",
-    "<|special_2632|>",
-    "<|special_2633|>",
-    "<|special_2634|>",
-    "<|special_2635|>",
-    "<|special_2636|>",
-    "<|special_2637|>",
-    "<|special_2638|>",
-    "<|special_2639|>",
-    "<|special_2640|>",
-    "<|special_2641|>",
-    "<|special_2642|>",
-    "<|special_2643|>",
-    "<|special_2644|>",
-    "<|special_2645|>",
-    "<|special_2646|>",
-    "<|special_2647|>",
-    "<|special_2648|>",
-    "<|special_2649|>",
-    "<|special_2650|>",
-    "<|special_2651|>",
-    "<|special_2652|>",
-    "<|special_2653|>",
-    "<|special_2654|>",
-    "<|special_2655|>",
-    "<|special_2656|>",
-    "<|special_2657|>",
-    "<|special_2658|>",
-    "<|special_2659|>",
-    "<|special_2660|>",
-    "<|special_2661|>",
-    "<|special_2662|>",
-    "<|special_2663|>",
-    "<|special_2664|>",
-    "<|special_2665|>",
-    "<|special_2666|>",
-    "<|special_2667|>",
-    "<|special_2668|>",
-    "<|special_2669|>",
-    "<|special_2670|>",
-    "<|special_2671|>",
-    "<|special_2672|>",
-    "<|special_2673|>",
-    "<|special_2674|>",
-    "<|special_2675|>",
-    "<|special_2676|>",
-    "<|special_2677|>",
-    "<|special_2678|>",
-    "<|special_2679|>",
-    "<|special_2680|>",
-    "<|special_2681|>",
-    "<|special_2682|>",
-    "<|special_2683|>",
-    "<|special_2684|>",
-    "<|special_2685|>",
-    "<|special_2686|>",
-    "<|special_2687|>",
-    "<|special_2688|>",
-    "<|special_2689|>",
-    "<|special_2690|>",
-    "<|special_2691|>",
-    "<|special_2692|>",
-    "<|special_2693|>",
-    "<|special_2694|>",
-    "<|special_2695|>",
-    "<|special_2696|>",
-    "<|special_2697|>",
-    "<|special_2698|>",
-    "<|special_2699|>",
-    "<|special_2700|>",
-    "<|special_2701|>",
-    "<|special_2702|>",
-    "<|special_2703|>",
-    "<|special_2704|>",
-    "<|special_2705|>",
-    "<|special_2706|>",
-    "<|special_2707|>",
-    "<|special_2708|>",
-    "<|special_2709|>",
-    "<|special_2710|>",
-    "<|special_2711|>",
-    "<|special_2712|>",
-    "<|special_2713|>",
-    "<|special_2714|>",
-    "<|special_2715|>",
-    "<|special_2716|>",
-    "<|special_2717|>",
-    "<|special_2718|>",
-    "<|special_2719|>",
-    "<|special_2720|>",
-    "<|special_2721|>",
-    "<|special_2722|>",
-    "<|special_2723|>",
-    "<|special_2724|>",
-    "<|special_2725|>",
-    "<|special_2726|>",
-    "<|special_2727|>",
-    "<|special_2728|>",
-    "<|special_2729|>",
-    "<|special_2730|>",
-    "<|special_2731|>",
-    "<|special_2732|>",
-    "<|special_2733|>",
-    "<|special_2734|>",
-    "<|special_2735|>",
-    "<|special_2736|>",
-    "<|special_2737|>",
-    "<|special_2738|>",
-    "<|special_2739|>",
-    "<|special_2740|>",
-    "<|special_2741|>",
-    "<|special_2742|>",
-    "<|special_2743|>",
-    "<|special_2744|>",
-    "<|special_2745|>",
-    "<|special_2746|>",
-    "<|special_2747|>",
-    "<|special_2748|>",
-    "<|special_2749|>",
-    "<|special_2750|>",
-    "<|special_2751|>",
-    "<|special_2752|>",
-    "<|special_2753|>",
-    "<|special_2754|>",
-    "<|special_2755|>",
-    "<|special_2756|>",
-    "<|special_2757|>",
-    "<|special_2758|>",
-    "<|special_2759|>",
-    "<|special_2760|>",
-    "<|special_2761|>",
-    "<|special_2762|>",
-    "<|special_2763|>",
-    "<|special_2764|>",
-    "<|special_2765|>",
-    "<|special_2766|>",
-    "<|special_2767|>",
-    "<|special_2768|>",
-    "<|special_2769|>",
-    "<|special_2770|>",
-    "<|special_2771|>",
-    "<|special_2772|>",
-    "<|special_2773|>",
-    "<|special_2774|>",
-    "<|special_2775|>",
-    "<|special_2776|>",
-    "<|special_2777|>",
-    "<|special_2778|>",
-    "<|special_2779|>",
-    "<|special_2780|>",
-    "<|special_2781|>",
-    "<|special_2782|>",
-    "<|special_2783|>",
-    "<|special_2784|>",
-    "<|special_2785|>",
-    "<|special_2786|>",
-    "<|special_2787|>",
-    "<|special_2788|>",
-    "<|special_2789|>",
-    "<|special_2790|>",
-    "<|special_2791|>",
-    "<|special_2792|>",
-    "<|special_2793|>",
-    "<|special_2794|>",
-    "<|special_2795|>",
-    "<|special_2796|>",
-    "<|special_2797|>",
-    "<|special_2798|>",
-    "<|special_2799|>",
-    "<|special_2800|>",
-    "<|special_2801|>",
-    "<|special_2802|>",
-    "<|special_2803|>",
-    "<|special_2804|>",
-    "<|special_2805|>",
-    "<|special_2806|>",
-    "<|special_2807|>",
-    "<|special_2808|>",
-    "<|special_2809|>",
-    "<|special_2810|>",
-    "<|special_2811|>",
-    "<|special_2812|>",
-    "<|special_2813|>",
-    "<|special_2814|>",
-    "<|special_2815|>",
-    "<|special_2816|>",
-    "<|special_2817|>",
-    "<|special_2818|>",
-    "<|special_2819|>",
-    "<|special_2820|>",
-    "<|special_2821|>",
-    "<|special_2822|>",
-    "<|special_2823|>",
-    "<|special_2824|>",
-    "<|special_2825|>",
-    "<|special_2826|>",
-    "<|special_2827|>",
-    "<|special_2828|>",
-    "<|special_2829|>",
-    "<|special_2830|>",
-    "<|special_2831|>",
-    "<|special_2832|>",
-    "<|special_2833|>",
-    "<|special_2834|>",
-    "<|special_2835|>",
-    "<|special_2836|>",
-    "<|special_2837|>",
-    "<|special_2838|>",
-    "<|special_2839|>",
-    "<|special_2840|>",
-    "<|special_2841|>",
-    "<|special_2842|>",
-    "<|special_2843|>",
-    "<|special_2844|>",
-    "<|special_2845|>",
-    "<|special_2846|>",
-    "<|special_2847|>",
-    "<|special_2848|>",
-    "<|special_2849|>",
-    "<|special_2850|>",
-    "<|special_2851|>",
-    "<|special_2852|>",
-    "<|special_2853|>",
-    "<|special_2854|>",
-    "<|special_2855|>",
-    "<|special_2856|>",
-    "<|special_2857|>",
-    "<|special_2858|>",
-    "<|special_2859|>",
-    "<|special_2860|>",
-    "<|special_2861|>",
-    "<|special_2862|>",
-    "<|special_2863|>",
-    "<|special_2864|>",
-    "<|special_2865|>",
-    "<|special_2866|>",
-    "<|special_2867|>",
-    "<|special_2868|>",
-    "<|special_2869|>",
-    "<|special_2870|>",
-    "<|special_2871|>",
-    "<|special_2872|>",
-    "<|special_2873|>",
-    "<|special_2874|>",
-    "<|special_2875|>",
-    "<|special_2876|>",
-    "<|special_2877|>",
-    "<|special_2878|>",
-    "<|special_2879|>",
-    "<|special_2880|>",
-    "<|special_2881|>",
-    "<|special_2882|>",
-    "<|special_2883|>",
-    "<|special_2884|>",
-    "<|special_2885|>",
-    "<|special_2886|>",
-    "<|special_2887|>",
-    "<|special_2888|>",
-    "<|special_2889|>",
-    "<|special_2890|>",
-    "<|special_2891|>",
-    "<|special_2892|>",
-    "<|special_2893|>",
-    "<|special_2894|>",
-    "<|special_2895|>",
-    "<|special_2896|>",
-    "<|special_2897|>",
-    "<|special_2898|>",
-    "<|special_2899|>",
-    "<|special_2900|>",
-    "<|special_2901|>",
-    "<|special_2902|>",
-    "<|special_2903|>",
-    "<|special_2904|>",
-    "<|special_2905|>",
-    "<|special_2906|>",
-    "<|special_2907|>",
-    "<|special_2908|>",
-    "<|special_2909|>",
-    "<|special_2910|>",
-    "<|special_2911|>",
-    "<|special_2912|>",
-    "<|special_2913|>",
-    "<|special_2914|>",
-    "<|special_2915|>",
-    "<|special_2916|>",
-    "<|special_2917|>",
-    "<|special_2918|>",
-    "<|special_2919|>",
-    "<|special_2920|>",
-    "<|special_2921|>",
-    "<|special_2922|>",
-    "<|special_2923|>",
-    "<|special_2924|>",
-    "<|special_2925|>",
-    "<|special_2926|>",
-    "<|special_2927|>",
-    "<|special_2928|>",
-    "<|special_2929|>",
-    "<|special_2930|>",
-    "<|special_2931|>",
-    "<|special_2932|>",
-    "<|special_2933|>",
-    "<|special_2934|>",
-    "<|special_2935|>",
-    "<|special_2936|>",
-    "<|special_2937|>",
-    "<|special_2938|>",
-    "<|special_2939|>",
-    "<|special_2940|>",
-    "<|special_2941|>",
-    "<|special_2942|>",
-    "<|special_2943|>",
-    "<|special_2944|>",
-    "<|special_2945|>",
-    "<|special_2946|>",
-    "<|special_2947|>",
-    "<|special_2948|>",
-    "<|special_2949|>",
-    "<|special_2950|>",
-    "<|special_2951|>",
-    "<|special_2952|>",
-    "<|special_2953|>",
-    "<|special_2954|>",
-    "<|special_2955|>",
-    "<|special_2956|>",
-    "<|special_2957|>",
-    "<|special_2958|>",
-    "<|special_2959|>",
-    "<|special_2960|>",
-    "<|special_2961|>",
-    "<|special_2962|>",
-    "<|special_2963|>",
-    "<|special_2964|>",
-    "<|special_2965|>",
-    "<|special_2966|>",
-    "<|special_2967|>",
-    "<|special_2968|>",
-    "<|special_2969|>",
-    "<|special_2970|>",
-    "<|special_2971|>",
-    "<|special_2972|>",
-    "<|special_2973|>",
-    "<|special_2974|>",
-    "<|special_2975|>",
-    "<|special_2976|>",
-    "<|special_2977|>",
-    "<|special_2978|>",
-    "<|special_2979|>",
-    "<|special_2980|>",
-    "<|special_2981|>",
-    "<|special_2982|>",
-    "<|special_2983|>",
-    "<|special_2984|>",
-    "<|special_2985|>",
-    "<|special_2986|>",
-    "<|special_2987|>",
-    "<|special_2988|>",
-    "<|special_2989|>",
-    "<|special_2990|>",
-    "<|special_2991|>",
-    "<|special_2992|>",
-    "<|special_2993|>",
-    "<|special_2994|>",
-    "<|special_2995|>",
-    "<|special_2996|>",
-    "<|special_2997|>",
-    "<|special_2998|>",
-    "<|special_2999|>",
-    "<|special_3000|>",
-    "<|special_3001|>",
-    "<|special_3002|>",
-    "<|special_3003|>",
-    "<|special_3004|>",
-    "<|special_3005|>",
-    "<|special_3006|>",
-    "<|special_3007|>",
-    "<|special_3008|>",
-    "<|special_3009|>",
-    "<|special_3010|>",
-    "<|special_3011|>",
-    "<|special_3012|>",
-    "<|special_3013|>",
-    "<|special_3014|>",
-    "<|special_3015|>",
-    "<|special_3016|>",
-    "<|special_3017|>",
-    "<|special_3018|>",
-    "<|special_3019|>",
-    "<|special_3020|>",
-    "<|special_3021|>",
-    "<|special_3022|>",
-    "<|special_3023|>",
-    "<|special_3024|>",
-    "<|special_3025|>",
-    "<|special_3026|>",
-    "<|special_3027|>",
-    "<|special_3028|>",
-    "<|special_3029|>",
-    "<|special_3030|>",
-    "<|special_3031|>",
-    "<|special_3032|>",
-    "<|special_3033|>",
-    "<|special_3034|>",
-    "<|special_3035|>",
-    "<|special_3036|>",
-    "<|special_3037|>",
-    "<|special_3038|>",
-    "<|special_3039|>",
-    "<|special_3040|>",
-    "<|special_3041|>",
-    "<|special_3042|>",
-    "<|special_3043|>",
-    "<|special_3044|>",
-    "<|special_3045|>",
-    "<|special_3046|>",
-    "<|special_3047|>",
-    "<|special_3048|>",
-    "<|special_3049|>",
-    "<|special_3050|>",
-    "<|special_3051|>",
-    "<|special_3052|>",
-    "<|special_3053|>",
-    "<|special_3054|>",
-    "<|special_3055|>",
-    "<|special_3056|>",
-    "<|special_3057|>",
-    "<|special_3058|>",
-    "<|special_3059|>",
-    "<|special_3060|>",
-    "<|special_3061|>",
-    "<|special_3062|>",
-    "<|special_3063|>",
-    "<|special_3064|>",
-    "<|special_3065|>",
-    "<|special_3066|>",
-    "<|special_3067|>",
-    "<|special_3068|>",
-    "<|special_3069|>",
-    "<|special_3070|>",
-    "<|special_3071|>",
-    "<|special_3072|>",
-    "<|special_3073|>",
-    "<|special_3074|>",
-    "<|special_3075|>",
-    "<|special_3076|>",
-    "<|special_3077|>",
-    "<|special_3078|>",
-    "<|special_3079|>",
-    "<|special_3080|>",
-    "<|special_3081|>",
-    "<|special_3082|>",
-    "<|special_3083|>",
-    "<|special_3084|>",
-    "<|special_3085|>",
-    "<|special_3086|>",
-    "<|special_3087|>",
-    "<|special_3088|>",
-    "<|special_3089|>",
-    "<|special_3090|>",
-    "<|special_3091|>",
-    "<|special_3092|>",
-    "<|special_3093|>",
-    "<|special_3094|>",
-    "<|special_3095|>",
-    "<|special_3096|>",
-    "<|special_3097|>",
-    "<|special_3098|>",
-    "<|special_3099|>",
-    "<|special_3100|>",
-    "<|special_3101|>",
-    "<|special_3102|>",
-    "<|special_3103|>",
-    "<|special_3104|>",
-    "<|special_3105|>",
-    "<|special_3106|>",
-    "<|special_3107|>",
-    "<|special_3108|>",
-    "<|special_3109|>",
-    "<|special_3110|>",
-    "<|special_3111|>",
-    "<|special_3112|>",
-    "<|special_3113|>",
-    "<|special_3114|>",
-    "<|special_3115|>",
-    "<|special_3116|>",
-    "<|special_3117|>",
-    "<|special_3118|>",
-    "<|special_3119|>",
-    "<|special_3120|>",
-    "<|special_3121|>",
-    "<|special_3122|>",
-    "<|special_3123|>",
-    "<|special_3124|>",
-    "<|special_3125|>",
-    "<|special_3126|>",
-    "<|special_3127|>",
-    "<|special_3128|>",
-    "<|special_3129|>",
-    "<|special_3130|>",
-    "<|special_3131|>",
-    "<|special_3132|>",
-    "<|special_3133|>",
-    "<|special_3134|>",
-    "<|special_3135|>",
-    "<|special_3136|>",
-    "<|special_3137|>",
-    "<|special_3138|>",
-    "<|special_3139|>",
-    "<|special_3140|>",
-    "<|special_3141|>",
-    "<|special_3142|>",
-    "<|special_3143|>",
-    "<|special_3144|>",
-    "<|special_3145|>",
-    "<|special_3146|>",
-    "<|special_3147|>",
-    "<|special_3148|>",
-    "<|special_3149|>",
-    "<|special_3150|>",
-    "<|special_3151|>",
-    "<|special_3152|>",
-    "<|special_3153|>",
-    "<|special_3154|>",
-    "<|special_3155|>",
-    "<|special_3156|>",
-    "<|special_3157|>",
-    "<|special_3158|>",
-    "<|special_3159|>",
-    "<|special_3160|>",
-    "<|special_3161|>",
-    "<|special_3162|>",
-    "<|special_3163|>",
-    "<|special_3164|>",
-    "<|special_3165|>",
-    "<|special_3166|>",
-    "<|special_3167|>",
-    "<|special_3168|>",
-    "<|special_3169|>",
-    "<|special_3170|>",
-    "<|special_3171|>",
-    "<|special_3172|>",
-    "<|special_3173|>",
-    "<|special_3174|>",
-    "<|special_3175|>",
-    "<|special_3176|>",
-    "<|special_3177|>",
-    "<|special_3178|>",
-    "<|special_3179|>",
-    "<|special_3180|>",
-    "<|special_3181|>",
-    "<|special_3182|>",
-    "<|special_3183|>",
-    "<|special_3184|>",
-    "<|special_3185|>",
-    "<|special_3186|>",
-    "<|special_3187|>",
-    "<|special_3188|>",
-    "<|special_3189|>",
-    "<|special_3190|>",
-    "<|special_3191|>",
-    "<|special_3192|>",
-    "<|special_3193|>",
-    "<|special_3194|>",
-    "<|special_3195|>",
-    "<|special_3196|>",
-    "<|special_3197|>",
-    "<|special_3198|>",
-    "<|special_3199|>",
-    "<|special_3200|>",
-    "<|special_3201|>",
-    "<|special_3202|>",
-    "<|special_3203|>",
-    "<|special_3204|>",
-    "<|special_3205|>",
-    "<|special_3206|>",
-    "<|special_3207|>",
-    "<|special_3208|>",
-    "<|special_3209|>",
-    "<|special_3210|>",
-    "<|special_3211|>",
-    "<|special_3212|>",
-    "<|special_3213|>",
-    "<|special_3214|>",
-    "<|special_3215|>",
-    "<|special_3216|>",
-    "<|special_3217|>",
-    "<|special_3218|>",
-    "<|special_3219|>",
-    "<|special_3220|>",
-    "<|special_3221|>",
-    "<|special_3222|>",
-    "<|special_3223|>",
-    "<|special_3224|>",
-    "<|special_3225|>",
-    "<|special_3226|>",
-    "<|special_3227|>",
-    "<|special_3228|>",
-    "<|special_3229|>",
-    "<|special_3230|>",
-    "<|special_3231|>",
-    "<|special_3232|>",
-    "<|special_3233|>",
-    "<|special_3234|>",
-    "<|special_3235|>",
-    "<|special_3236|>",
-    "<|special_3237|>",
-    "<|special_3238|>",
-    "<|special_3239|>",
-    "<|special_3240|>",
-    "<|special_3241|>",
-    "<|special_3242|>",
-    "<|special_3243|>",
-    "<|special_3244|>",
-    "<|special_3245|>",
-    "<|special_3246|>",
-    "<|special_3247|>",
-    "<|special_3248|>",
-    "<|special_3249|>",
-    "<|special_3250|>",
-    "<|special_3251|>",
-    "<|special_3252|>",
-    "<|special_3253|>",
-    "<|special_3254|>",
-    "<|special_3255|>",
-    "<|special_3256|>",
-    "<|special_3257|>",
-    "<|special_3258|>",
-    "<|special_3259|>",
-    "<|special_3260|>",
-    "<|special_3261|>",
-    "<|special_3262|>",
-    "<|special_3263|>",
-    "<|special_3264|>",
-    "<|special_3265|>",
-    "<|special_3266|>",
-    "<|special_3267|>",
-    "<|special_3268|>",
-    "<|special_3269|>",
-    "<|special_3270|>",
-    "<|special_3271|>",
-    "<|special_3272|>",
-    "<|special_3273|>",
-    "<|special_3274|>",
-    "<|special_3275|>",
-    "<|special_3276|>",
-    "<|special_3277|>",
-    "<|special_3278|>",
-    "<|special_3279|>",
-    "<|special_3280|>",
-    "<|special_3281|>",
-    "<|special_3282|>",
-    "<|special_3283|>",
-    "<|special_3284|>",
-    "<|special_3285|>",
-    "<|special_3286|>",
-    "<|special_3287|>",
-    "<|special_3288|>",
-    "<|special_3289|>",
-    "<|special_3290|>",
-    "<|special_3291|>",
-    "<|special_3292|>",
-    "<|special_3293|>",
-    "<|special_3294|>",
-    "<|special_3295|>",
-    "<|special_3296|>",
-    "<|special_3297|>",
-    "<|special_3298|>",
-    "<|special_3299|>",
-    "<|special_3300|>",
-    "<|special_3301|>",
-    "<|special_3302|>",
-    "<|special_3303|>",
-    "<|special_3304|>",
-    "<|special_3305|>",
-    "<|special_3306|>",
-    "<|special_3307|>",
-    "<|special_3308|>",
-    "<|special_3309|>",
-    "<|special_3310|>",
-    "<|special_3311|>",
-    "<|special_3312|>",
-    "<|special_3313|>",
-    "<|special_3314|>",
-    "<|special_3315|>",
-    "<|special_3316|>",
-    "<|special_3317|>",
-    "<|special_3318|>",
-    "<|special_3319|>",
-    "<|special_3320|>",
-    "<|special_3321|>",
-    "<|special_3322|>",
-    "<|special_3323|>",
-    "<|special_3324|>",
-    "<|special_3325|>",
-    "<|special_3326|>",
-    "<|special_3327|>",
-    "<|special_3328|>",
-    "<|special_3329|>",
-    "<|special_3330|>",
-    "<|special_3331|>",
-    "<|special_3332|>",
-    "<|special_3333|>",
-    "<|special_3334|>",
-    "<|special_3335|>",
-    "<|special_3336|>",
-    "<|special_3337|>",
-    "<|special_3338|>",
-    "<|special_3339|>",
-    "<|special_3340|>",
-    "<|special_3341|>",
-    "<|special_3342|>",
-    "<|special_3343|>",
-    "<|special_3344|>",
-    "<|special_3345|>",
-    "<|special_3346|>",
-    "<|special_3347|>",
-    "<|special_3348|>",
-    "<|special_3349|>",
-    "<|special_3350|>",
-    "<|special_3351|>",
-    "<|special_3352|>",
-    "<|special_3353|>",
-    "<|special_3354|>",
-    "<|special_3355|>",
-    "<|special_3356|>",
-    "<|special_3357|>",
-    "<|special_3358|>",
-    "<|special_3359|>",
-    "<|special_3360|>",
-    "<|special_3361|>",
-    "<|special_3362|>",
-    "<|special_3363|>",
-    "<|special_3364|>",
-    "<|special_3365|>",
-    "<|special_3366|>",
-    "<|special_3367|>",
-    "<|special_3368|>",
-    "<|special_3369|>",
-    "<|special_3370|>",
-    "<|special_3371|>",
-    "<|special_3372|>",
-    "<|special_3373|>",
-    "<|special_3374|>",
-    "<|special_3375|>",
-    "<|special_3376|>",
-    "<|special_3377|>",
-    "<|special_3378|>",
-    "<|special_3379|>",
-    "<|special_3380|>",
-    "<|special_3381|>",
-    "<|special_3382|>",
-    "<|special_3383|>",
-    "<|special_3384|>",
-    "<|special_3385|>",
-    "<|special_3386|>",
-    "<|special_3387|>",
-    "<|special_3388|>",
-    "<|special_3389|>",
-    "<|special_3390|>",
-    "<|special_3391|>",
-    "<|special_3392|>",
-    "<|special_3393|>",
-    "<|special_3394|>",
-    "<|special_3395|>",
-    "<|special_3396|>",
-    "<|special_3397|>",
-    "<|special_3398|>",
-    "<|special_3399|>",
-    "<|special_3400|>",
-    "<|special_3401|>",
-    "<|special_3402|>",
-    "<|special_3403|>",
-    "<|special_3404|>",
-    "<|special_3405|>",
-    "<|special_3406|>",
-    "<|special_3407|>",
-    "<|special_3408|>",
-    "<|special_3409|>",
-    "<|special_3410|>",
-    "<|special_3411|>",
-    "<|special_3412|>",
-    "<|special_3413|>",
-    "<|special_3414|>",
-    "<|special_3415|>",
-    "<|special_3416|>",
-    "<|special_3417|>",
-    "<|special_3418|>",
-    "<|special_3419|>",
-    "<|special_3420|>",
-    "<|special_3421|>",
-    "<|special_3422|>",
-    "<|special_3423|>",
-    "<|special_3424|>",
-    "<|special_3425|>",
-    "<|special_3426|>",
-    "<|special_3427|>",
-    "<|special_3428|>",
-    "<|special_3429|>",
-    "<|special_3430|>",
-    "<|special_3431|>",
-    "<|special_3432|>",
-    "<|special_3433|>",
-    "<|special_3434|>",
-    "<|special_3435|>",
-    "<|special_3436|>",
-    "<|special_3437|>",
-    "<|special_3438|>",
-    "<|special_3439|>",
-    "<|special_3440|>",
-    "<|special_3441|>",
-    "<|special_3442|>",
-    "<|special_3443|>",
-    "<|special_3444|>",
-    "<|special_3445|>",
-    "<|special_3446|>",
-    "<|special_3447|>",
-    "<|special_3448|>",
-    "<|special_3449|>",
-    "<|special_3450|>",
-    "<|special_3451|>",
-    "<|special_3452|>",
-    "<|special_3453|>",
-    "<|special_3454|>",
-    "<|special_3455|>",
-    "<|special_3456|>",
-    "<|special_3457|>",
-    "<|special_3458|>",
-    "<|special_3459|>",
-    "<|special_3460|>",
-    "<|special_3461|>",
-    "<|special_3462|>",
-    "<|special_3463|>",
-    "<|special_3464|>",
-    "<|special_3465|>",
-    "<|special_3466|>",
-    "<|special_3467|>",
-    "<|special_3468|>",
-    "<|special_3469|>",
-    "<|special_3470|>",
-    "<|special_3471|>",
-    "<|special_3472|>",
-    "<|special_3473|>",
-    "<|special_3474|>",
-    "<|special_3475|>",
-    "<|special_3476|>",
-    "<|special_3477|>",
-    "<|special_3478|>",
-    "<|special_3479|>",
-    "<|special_3480|>",
-    "<|special_3481|>",
-    "<|special_3482|>",
-    "<|special_3483|>",
-    "<|special_3484|>",
-    "<|special_3485|>",
-    "<|special_3486|>",
-    "<|special_3487|>",
-    "<|special_3488|>",
-    "<|special_3489|>",
-    "<|special_3490|>",
-    "<|special_3491|>",
-    "<|special_3492|>",
-    "<|special_3493|>",
-    "<|special_3494|>",
-    "<|special_3495|>",
-    "<|special_3496|>",
-    "<|special_3497|>",
-    "<|special_3498|>",
-    "<|special_3499|>",
-    "<|special_3500|>",
-    "<|special_3501|>",
-    "<|special_3502|>",
-    "<|special_3503|>",
-    "<|special_3504|>",
-    "<|special_3505|>",
-    "<|special_3506|>",
-    "<|special_3507|>",
-    "<|special_3508|>",
-    "<|special_3509|>",
-    "<|special_3510|>",
-    "<|special_3511|>",
-    "<|special_3512|>",
-    "<|special_3513|>",
-    "<|special_3514|>",
-    "<|special_3515|>",
-    "<|special_3516|>",
-    "<|special_3517|>",
-    "<|special_3518|>",
-    "<|special_3519|>",
-    "<|special_3520|>",
-    "<|special_3521|>",
-    "<|special_3522|>",
-    "<|special_3523|>",
-    "<|special_3524|>",
-    "<|special_3525|>",
-    "<|special_3526|>",
-    "<|special_3527|>",
-    "<|special_3528|>",
-    "<|special_3529|>",
-    "<|special_3530|>",
-    "<|special_3531|>",
-    "<|special_3532|>",
-    "<|special_3533|>",
-    "<|special_3534|>",
-    "<|special_3535|>",
-    "<|special_3536|>",
-    "<|special_3537|>",
-    "<|special_3538|>",
-    "<|special_3539|>",
-    "<|special_3540|>",
-    "<|special_3541|>",
-    "<|special_3542|>",
-    "<|special_3543|>",
-    "<|special_3544|>",
-    "<|special_3545|>",
-    "<|special_3546|>",
-    "<|special_3547|>",
-    "<|special_3548|>",
-    "<|special_3549|>",
-    "<|special_3550|>",
-    "<|special_3551|>",
-    "<|special_3552|>",
-    "<|special_3553|>",
-    "<|special_3554|>",
-    "<|special_3555|>",
-    "<|special_3556|>",
-    "<|special_3557|>",
-    "<|special_3558|>",
-    "<|special_3559|>",
-    "<|special_3560|>",
-    "<|special_3561|>",
-    "<|special_3562|>",
-    "<|special_3563|>",
-    "<|special_3564|>",
-    "<|special_3565|>",
-    "<|special_3566|>",
-    "<|special_3567|>",
-    "<|special_3568|>",
-    "<|special_3569|>",
-    "<|special_3570|>",
-    "<|special_3571|>",
-    "<|special_3572|>",
-    "<|special_3573|>",
-    "<|special_3574|>",
-    "<|special_3575|>",
-    "<|special_3576|>",
-    "<|special_3577|>",
-    "<|special_3578|>",
-    "<|special_3579|>",
-    "<|special_3580|>",
-    "<|special_3581|>",
-    "<|special_3582|>",
-    "<|special_3583|>",
-    "<|special_3584|>",
-    "<|special_3585|>",
-    "<|special_3586|>",
-    "<|special_3587|>",
-    "<|special_3588|>",
-    "<|special_3589|>",
-    "<|special_3590|>",
-    "<|special_3591|>",
-    "<|special_3592|>",
-    "<|special_3593|>",
-    "<|special_3594|>",
-    "<|special_3595|>",
-    "<|special_3596|>",
-    "<|special_3597|>",
-    "<|special_3598|>",
-    "<|special_3599|>",
-    "<|special_3600|>",
-    "<|special_3601|>",
-    "<|special_3602|>",
-    "<|special_3603|>",
-    "<|special_3604|>",
-    "<|special_3605|>",
-    "<|special_3606|>",
-    "<|special_3607|>",
-    "<|special_3608|>",
-    "<|special_3609|>",
-    "<|special_3610|>",
-    "<|special_3611|>",
-    "<|special_3612|>",
-    "<|special_3613|>",
-    "<|special_3614|>",
-    "<|special_3615|>",
-    "<|special_3616|>",
-    "<|special_3617|>",
-    "<|special_3618|>",
-    "<|special_3619|>",
-    "<|special_3620|>",
-    "<|special_3621|>",
-    "<|special_3622|>",
-    "<|special_3623|>",
-    "<|special_3624|>",
-    "<|special_3625|>",
-    "<|special_3626|>",
-    "<|special_3627|>",
-    "<|special_3628|>",
-    "<|special_3629|>",
-    "<|special_3630|>",
-    "<|special_3631|>",
-    "<|special_3632|>",
-    "<|special_3633|>",
-    "<|special_3634|>",
-    "<|special_3635|>",
-    "<|special_3636|>",
-    "<|special_3637|>",
-    "<|special_3638|>",
-    "<|special_3639|>",
-    "<|special_3640|>",
-    "<|special_3641|>",
-    "<|special_3642|>",
-    "<|special_3643|>",
-    "<|special_3644|>",
-    "<|special_3645|>",
-    "<|special_3646|>",
-    "<|special_3647|>",
-    "<|special_3648|>",
-    "<|special_3649|>",
-    "<|special_3650|>",
-    "<|special_3651|>",
-    "<|special_3652|>",
-    "<|special_3653|>",
-    "<|special_3654|>",
-    "<|special_3655|>",
-    "<|special_3656|>",
-    "<|special_3657|>",
-    "<|special_3658|>",
-    "<|special_3659|>",
-    "<|special_3660|>",
-    "<|special_3661|>",
-    "<|special_3662|>",
-    "<|special_3663|>",
-    "<|special_3664|>",
-    "<|special_3665|>",
-    "<|special_3666|>",
-    "<|special_3667|>",
-    "<|special_3668|>",
-    "<|special_3669|>",
-    "<|special_3670|>",
-    "<|special_3671|>",
-    "<|special_3672|>",
-    "<|special_3673|>",
-    "<|special_3674|>",
-    "<|special_3675|>",
-    "<|special_3676|>",
-    "<|special_3677|>",
-    "<|special_3678|>",
-    "<|special_3679|>",
-    "<|special_3680|>",
-    "<|special_3681|>",
-    "<|special_3682|>",
-    "<|special_3683|>",
-    "<|special_3684|>",
-    "<|special_3685|>",
-    "<|special_3686|>",
-    "<|special_3687|>",
-    "<|special_3688|>",
-    "<|special_3689|>",
-    "<|special_3690|>",
-    "<|special_3691|>",
-    "<|special_3692|>",
-    "<|special_3693|>",
-    "<|special_3694|>",
-    "<|special_3695|>",
-    "<|special_3696|>",
-    "<|special_3697|>",
-    "<|special_3698|>",
-    "<|special_3699|>",
-    "<|special_3700|>",
-    "<|special_3701|>",
-    "<|special_3702|>",
-    "<|special_3703|>",
-    "<|special_3704|>",
-    "<|special_3705|>",
-    "<|special_3706|>",
-    "<|special_3707|>",
-    "<|special_3708|>",
-    "<|special_3709|>",
-    "<|special_3710|>",
-    "<|special_3711|>",
-    "<|special_3712|>",
-    "<|special_3713|>",
-    "<|special_3714|>",
-    "<|special_3715|>",
-    "<|special_3716|>",
-    "<|special_3717|>",
-    "<|special_3718|>",
-    "<|special_3719|>",
-    "<|special_3720|>",
-    "<|special_3721|>",
-    "<|special_3722|>",
-    "<|special_3723|>",
-    "<|special_3724|>",
-    "<|special_3725|>",
-    "<|special_3726|>",
-    "<|special_3727|>",
-    "<|special_3728|>",
-    "<|special_3729|>",
-    "<|special_3730|>",
-    "<|special_3731|>",
-    "<|special_3732|>",
-    "<|special_3733|>",
-    "<|special_3734|>",
-    "<|special_3735|>",
-    "<|special_3736|>",
-    "<|special_3737|>",
-    "<|special_3738|>",
-    "<|special_3739|>",
-    "<|special_3740|>",
-    "<|special_3741|>",
-    "<|special_3742|>",
-    "<|special_3743|>",
-    "<|special_3744|>",
-    "<|special_3745|>",
-    "<|special_3746|>",
-    "<|special_3747|>",
-    "<|special_3748|>",
-    "<|special_3749|>",
-    "<|special_3750|>",
-    "<|special_3751|>",
-    "<|special_3752|>",
-    "<|special_3753|>",
-    "<|special_3754|>",
-    "<|special_3755|>",
-    "<|special_3756|>",
-    "<|special_3757|>",
-    "<|special_3758|>",
-    "<|special_3759|>",
-    "<|special_3760|>",
-    "<|special_3761|>",
-    "<|special_3762|>",
-    "<|special_3763|>",
-    "<|special_3764|>",
-    "<|special_3765|>",
-    "<|special_3766|>",
-    "<|special_3767|>",
-    "<|special_3768|>",
-    "<|special_3769|>",
-    "<|special_3770|>",
-    "<|special_3771|>",
-    "<|special_3772|>",
-    "<|special_3773|>",
-    "<|special_3774|>",
-    "<|special_3775|>",
-    "<|special_3776|>",
-    "<|special_3777|>",
-    "<|special_3778|>",
-    "<|special_3779|>",
-    "<|special_3780|>",
-    "<|special_3781|>",
-    "<|special_3782|>",
-    "<|special_3783|>",
-    "<|special_3784|>",
-    "<|special_3785|>",
-    "<|special_3786|>",
-    "<|special_3787|>",
-    "<|special_3788|>",
-    "<|special_3789|>",
-    "<|special_3790|>",
-    "<|special_3791|>",
-    "<|special_3792|>",
-    "<|special_3793|>",
-    "<|special_3794|>",
-    "<|special_3795|>",
-    "<|special_3796|>",
-    "<|special_3797|>",
-    "<|special_3798|>",
-    "<|special_3799|>",
-    "<|special_3800|>",
-    "<|special_3801|>",
-    "<|special_3802|>",
-    "<|special_3803|>",
-    "<|special_3804|>",
-    "<|special_3805|>",
-    "<|special_3806|>",
-    "<|special_3807|>",
-    "<|special_3808|>",
-    "<|special_3809|>",
-    "<|special_3810|>",
-    "<|special_3811|>",
-    "<|special_3812|>",
-    "<|special_3813|>",
-    "<|special_3814|>",
-    "<|special_3815|>",
-    "<|special_3816|>",
-    "<|special_3817|>",
-    "<|special_3818|>",
-    "<|special_3819|>",
-    "<|special_3820|>",
-    "<|special_3821|>",
-    "<|special_3822|>",
-    "<|special_3823|>",
-    "<|special_3824|>",
-    "<|special_3825|>",
-    "<|special_3826|>",
-    "<|special_3827|>",
-    "<|special_3828|>",
-    "<|special_3829|>",
-    "<|special_3830|>",
-    "<|special_3831|>",
-    "<|special_3832|>",
-    "<|special_3833|>",
-    "<|special_3834|>",
-    "<|special_3835|>",
-    "<|special_3836|>",
-    "<|special_3837|>",
-    "<|special_3838|>",
-    "<|special_3839|>",
-    "<|special_3840|>",
-    "<|special_3841|>",
-    "<|special_3842|>",
-    "<|special_3843|>",
-    "<|special_3844|>",
-    "<|special_3845|>",
-    "<|special_3846|>",
-    "<|special_3847|>",
-    "<|special_3848|>",
-    "<|special_3849|>",
-    "<|special_3850|>",
-    "<|special_3851|>",
-    "<|special_3852|>",
-    "<|special_3853|>",
-    "<|special_3854|>",
-    "<|special_3855|>",
-    "<|special_3856|>",
-    "<|special_3857|>",
-    "<|special_3858|>",
-    "<|special_3859|>",
-    "<|special_3860|>",
-    "<|special_3861|>",
-    "<|special_3862|>",
-    "<|special_3863|>",
-    "<|special_3864|>",
-    "<|special_3865|>",
-    "<|special_3866|>",
-    "<|special_3867|>",
-    "<|special_3868|>",
-    "<|special_3869|>",
-    "<|special_3870|>",
-    "<|special_3871|>",
-    "<|special_3872|>",
-    "<|special_3873|>",
-    "<|special_3874|>",
-    "<|special_3875|>",
-    "<|special_3876|>",
-    "<|special_3877|>",
-    "<|special_3878|>",
-    "<|special_3879|>",
-    "<|special_3880|>",
-    "<|special_3881|>",
-    "<|special_3882|>",
-    "<|special_3883|>",
-    "<|special_3884|>",
-    "<|special_3885|>",
-    "<|special_3886|>",
-    "<|special_3887|>",
-    "<|special_3888|>",
-    "<|special_3889|>",
-    "<|special_3890|>",
-    "<|special_3891|>",
-    "<|special_3892|>",
-    "<|special_3893|>",
-    "<|special_3894|>",
-    "<|special_3895|>",
-    "<|special_3896|>",
-    "<|special_3897|>",
-    "<|special_3898|>",
-    "<|special_3899|>",
-    "<|special_3900|>",
-    "<|special_3901|>",
-    "<|special_3902|>",
-    "<|special_3903|>",
-    "<|special_3904|>",
-    "<|special_3905|>",
-    "<|special_3906|>",
-    "<|special_3907|>",
-    "<|special_3908|>",
-    "<|special_3909|>",
-    "<|special_3910|>",
-    "<|special_3911|>",
-    "<|special_3912|>",
-    "<|special_3913|>",
-    "<|special_3914|>",
-    "<|special_3915|>",
-    "<|special_3916|>",
-    "<|special_3917|>",
-    "<|special_3918|>",
-    "<|special_3919|>",
-    "<|special_3920|>",
-    "<|special_3921|>",
-    "<|special_3922|>",
-    "<|special_3923|>",
-    "<|special_3924|>",
-    "<|special_3925|>",
-    "<|special_3926|>",
-    "<|special_3927|>",
-    "<|special_3928|>",
-    "<|special_3929|>",
-    "<|special_3930|>",
-    "<|special_3931|>",
-    "<|special_3932|>",
-    "<|special_3933|>",
-    "<|special_3934|>",
-    "<|special_3935|>",
-    "<|special_3936|>",
-    "<|special_3937|>",
-    "<|special_3938|>",
-    "<|special_3939|>",
-    "<|special_3940|>",
-    "<|special_3941|>",
-    "<|special_3942|>",
-    "<|special_3943|>",
-    "<|special_3944|>",
-    "<|special_3945|>",
-    "<|special_3946|>",
-    "<|special_3947|>",
-    "<|special_3948|>",
-    "<|special_3949|>",
-    "<|special_3950|>",
-    "<|special_3951|>",
-    "<|special_3952|>",
-    "<|special_3953|>",
-    "<|special_3954|>",
-    "<|special_3955|>",
-    "<|special_3956|>",
-    "<|special_3957|>",
-    "<|special_3958|>",
-    "<|special_3959|>",
-    "<|special_3960|>",
-    "<|special_3961|>",
-    "<|special_3962|>",
-    "<|special_3963|>",
-    "<|special_3964|>",
-    "<|special_3965|>",
-    "<|special_3966|>",
-    "<|special_3967|>",
-    "<|special_3968|>",
-    "<|special_3969|>",
-    "<|special_3970|>",
-    "<|special_3971|>",
-    "<|special_3972|>",
-    "<|special_3973|>",
-    "<|special_3974|>",
-    "<|special_3975|>",
-    "<|special_3976|>",
-    "<|special_3977|>",
-    "<|special_3978|>",
-    "<|special_3979|>",
-    "<|special_3980|>",
-    "<|special_3981|>",
-    "<|special_3982|>",
-    "<|special_3983|>",
-    "<|special_3984|>",
-    "<|special_3985|>",
-    "<|special_3986|>",
-    "<|special_3987|>",
-    "<|special_3988|>",
-    "<|special_3989|>",
-    "<|special_3990|>",
-    "<|special_3991|>",
-    "<|special_3992|>",
-    "<|special_3993|>",
-    "<|special_3994|>",
-    "<|special_3995|>",
-    "<|special_3996|>",
-    "<|special_3997|>",
-    "<|special_3998|>",
-    "<|special_3999|>",
-    "<|special_4000|>",
-    "<|special_4001|>",
-    "<|special_4002|>",
-    "<|special_4003|>",
-    "<|special_4004|>",
-    "<|special_4005|>",
-    "<|special_4006|>",
-    "<|special_4007|>",
-    "<|special_4008|>",
-    "<|special_4009|>",
-    "<|special_4010|>",
-    "<|special_4011|>",
-    "<|special_4012|>",
-    "<|special_4013|>",
-    "<|special_4014|>",
-    "<|special_4015|>",
-    "<|special_4016|>",
-    "<|special_4017|>",
-    "<|special_4018|>",
-    "<|special_4019|>",
-    "<|special_4020|>",
-    "<|special_4021|>",
-    "<|special_4022|>",
-    "<|special_4023|>",
-    "<|special_4024|>",
-    "<|special_4025|>",
-    "<|special_4026|>",
-    "<|special_4027|>",
-    "<|special_4028|>",
-    "<|special_4029|>",
-    "<|special_4030|>",
-    "<|special_4031|>",
-    "<|special_4032|>",
-    "<|special_4033|>",
-    "<|special_4034|>",
-    "<|special_4035|>",
-    "<|special_4036|>",
-    "<|special_4037|>",
-    "<|special_4038|>",
-    "<|special_4039|>",
-    "<|special_4040|>",
-    "<|special_4041|>",
-    "<|special_4042|>",
-    "<|special_4043|>",
-    "<|special_4044|>",
-    "<|special_4045|>",
-    "<|special_4046|>",
-    "<|special_4047|>",
-    "<|special_4048|>",
-    "<|special_4049|>",
-    "<|special_4050|>",
-    "<|special_4051|>",
-    "<|special_4052|>",
-    "<|special_4053|>",
-    "<|special_4054|>",
-    "<|special_4055|>",
-    "<|special_4056|>",
-    "<|special_4057|>",
-    "<|special_4058|>",
-    "<|special_4059|>",
-    "<|special_4060|>",
-    "<|special_4061|>",
-    "<|special_4062|>",
-    "<|special_4063|>",
-    "<|special_4064|>",
-    "<|special_4065|>",
-    "<|special_4066|>",
-    "<|special_4067|>",
-    "<|special_4068|>",
-    "<|special_4069|>",
-    "<|special_4070|>",
-    "<|special_4071|>",
-    "<|special_4072|>",
-    "<|special_4073|>",
-    "<|special_4074|>",
-    "<|special_4075|>",
-    "<|special_4076|>",
-    "<|special_4077|>",
-    "<|special_4078|>",
-    "<|special_4079|>",
-    "<|special_4080|>",
-    "<|special_4081|>",
-    "<|special_4082|>",
-    "<|special_4083|>",
-    "<|special_4084|>",
-    "<|special_4085|>",
-    "<|special_4086|>",
-    "<|special_4087|>",
-    "<|special_4088|>",
-    "<|special_4089|>",
-    "<|special_4090|>",
-    "<|special_4091|>",
-    "<|special_4092|>",
-    "<|special_4093|>",
-    "<|special_4094|>",
-    "<|special_4095|>"
-  ],
-  "bos_token": {
-    "content": "<|startoftext|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "eos_token": {
-    "content": "<|endoftext|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "pad_token": {
-    "content": "<|endoftext|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "unk_token": {
-    "content": "<unk>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  }
-}
+  "bos_token": "<|startoftext|>",
+  "cls_token": "[CLS]",
+  "eos_token": "<|endoftext|>",
+  "gmask_token": "[gMASK]",
+  "pad_token": "<|endoftext|>",
+  "mask_token": "<|mask|>"
+}

tokenization_llada2.py

@@ -0,0 +1,86 @@
+import os
+import logging
+from typing import Any, Iterator, Union
+from transformers import PreTrainedTokenizerFast
+from transformers.convert_slow_tokenizer import bytes_to_unicode
+
+from .tool_declaration_ts import encode_tools_to_typescript_style
+
+logger = logging.getLogger(__name__)
+
+
+def deep_sort_dict(obj: Any) -> Any:
+    """Deep sort dict keys recursively to ensure stable hashing and tokenization."""
+    if isinstance(obj, dict):
+        return {k: deep_sort_dict(v) for k, v in sorted(obj.items())}
+    if isinstance(obj, list):
+        return [deep_sort_dict(item) for item in obj]
+    return obj
+
+
+class CustomFastTokenizer(PreTrainedTokenizerFast):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+        # Byte-to-unicode mapping for downstream tasks requiring single-byte decoding
+        self.byte_encoder = bytes_to_unicode()
+        self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
+
+    @staticmethod
+    def _split_whitespaces_or_nonwhitespaces(
+        s: str, max_consecutive_slice_len: int
+    ) -> Iterator[str]:
+        current_slice_len = 0
+        current_slice_is_space = s[0].isspace() if len(s) > 0 else False
+        slice_start = 0
+
+        for i in range(len(s)):
+            is_now_space = s[i].isspace()
+            if current_slice_is_space ^ is_now_space:
+                current_slice_len = 1
+                current_slice_is_space = is_now_space
+            else:
+                current_slice_len += 1
+                if current_slice_len > max_consecutive_slice_len:
+                    yield s[slice_start:i]
+                    slice_start = i
+                    current_slice_len = 1
+        yield s[slice_start:]
+
+    def encode(self, text: Union[str, Any], *args, **kwargs) -> list[int]:
+        if not isinstance(text, str) or args or kwargs:
+            return super().encode(text, *args, **kwargs)
+
+        # Chunking thresholds to prevent OOM on very long texts
+        MAX_ENCODE_CHARS = 400_000
+        MAX_NO_WHITESPACES_CHARS = 25_000
+
+        all_substrs = []
+        for i in range(0, len(text), MAX_ENCODE_CHARS):
+            chunk = text[i : i + MAX_ENCODE_CHARS]
+            all_substrs.extend(
+                self._split_whitespaces_or_nonwhitespaces(
+                    chunk, MAX_NO_WHITESPACES_CHARS
+                )
+            )
+
+        t = []
+        for substr in all_substrs:
+            t.extend(super().encode(substr, add_special_tokens=False))
+
+        return t
+
+    def apply_chat_template(self, conversation, tools=None, **kwargs):
+        tools = deep_sort_dict(tools)
+
+        if tools:
+            try:
+                tools_ts_str = encode_tools_to_typescript_style(tools)
+                kwargs["tools_ts_str"] = tools_ts_str
+            except Exception as e:
+                logger.error(f"Failed to convert tools to TypeScript style: {e}")
+
+        return super().apply_chat_template(
+            conversation=conversation, tools=tools, **kwargs
+        )

tool_declaration_ts.py

@@ -0,0 +1,499 @@
+"""
+Encode structured tool declaration to typescript style string.
+"""
+
+import dataclasses
+import json
+import logging
+from collections.abc import Sequence
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+_TS_INDENT = "  "
+_TS_FIELD_DELIMITER = ",\n"
+
+
+class _SchemaRegistry:
+    """Registry for schema definitions to handle $ref resolution"""
+
+    def __init__(self):
+        self.definitions = {}
+        self.has_self_ref = False
+
+    def register_definitions(self, defs: dict[str, Any]):
+        """Register schema definitions from $defs section"""
+        if not defs:
+            return
+        for def_name, def_schema in defs.items():
+            self.definitions[def_name] = def_schema
+
+    def resolve_ref(self, ref: str) -> dict[str, Any]:
+        """Resolve a reference to its schema definition"""
+        if ref == "#":
+            self.has_self_ref = True
+            return {"$self_ref": True}
+        elif ref.startswith("#/$defs/"):
+            def_name = ref.split("/")[-1]
+            if def_name not in self.definitions:
+                raise ValueError(f"Reference not found: {ref}")
+            return self.definitions[def_name]
+        else:
+            raise ValueError(f"Unsupported reference format: {ref}")
+
+
+def _format_description(description: str, indent: str = "") -> str:
+    return "\n".join(
+        [f"{indent}// {line}" if line else "" for line in description.split("\n")]
+    )
+
+
+class _BaseType:
+    description: str
+    constraints: dict[str, Any]
+
+    def __init__(
+        self,
+        extra_props: dict[str, Any],
+        *,
+        allowed_constraint_keys: Sequence[str] = (),
+    ):
+        self.description = extra_props.get("description", "")
+        self.constraints = {
+            k: v for k, v in extra_props.items() if k in allowed_constraint_keys
+        }
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        raise NotImplementedError
+
+    def format_docstring(self, indent: str) -> str:
+        lines = []
+        if self.description:
+            lines.append(_format_description(self.description, indent))
+        if self.constraints:
+            constraints_str = ", ".join(
+                f"{k}: {v}"
+                for k, v in sorted(self.constraints.items(), key=lambda kv: kv[0])
+            )
+            lines.append(f"{indent}// {constraints_str}")
+
+        return "".join(x + "\n" for x in lines)
+
+
+class _ParameterTypeScalar(_BaseType):
+    type: str
+
+    def __init__(self, type: str, extra_props: dict[str, Any] | None = None):
+        self.type = type
+
+        allowed_constraint_keys: list[str] = []
+        if self.type == "string":
+            allowed_constraint_keys = ["maxLength", "minLength", "pattern"]
+        elif self.type in ("number", "integer"):
+            allowed_constraint_keys = ["maximum", "minimum"]
+
+        super().__init__(
+            extra_props or {}, allowed_constraint_keys=allowed_constraint_keys
+        )
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        # Map integer to number in TypeScript
+        if self.type == "integer":
+            return "number"
+        return self.type
+
+
+class _ParameterTypeObject(_BaseType):
+    properties: list["_Parameter"]
+    additional_properties: Any | None = None
+
+    def __init__(
+        self,
+        json_schema_object: dict[str, Any],
+        registry: _SchemaRegistry | None = None,
+    ):
+        super().__init__(json_schema_object)
+
+        self.properties = []
+        self.additional_properties = None
+
+        if not json_schema_object:
+            return
+
+        if "$defs" in json_schema_object and registry:
+            registry.register_definitions(json_schema_object["$defs"])
+
+        self.additional_properties = json_schema_object.get("additionalProperties")
+        if isinstance(self.additional_properties, dict):
+            self.additional_properties = _parse_parameter_type(
+                self.additional_properties, registry
+            )
+
+        if "properties" not in json_schema_object:
+            return
+
+        required_parameters = json_schema_object.get("required", [])
+        optional_parameters = set(json_schema_object["properties"].keys()) - set(
+            required_parameters
+        )
+
+        self.properties = [
+            _Parameter(
+                name=name,
+                type=_parse_parameter_type(prop, registry),
+                optional=name in optional_parameters,
+                default=prop.get("default") if isinstance(prop, dict) else None,
+            )
+            for name, prop in json_schema_object["properties"].items()
+        ]
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        # sort by optional, make the required parameters first
+        parameters = [p for p in self.properties if not p.optional]
+        opt_params = [p for p in self.properties if p.optional]
+
+        parameters = sorted(parameters, key=lambda p: p.name)
+        parameters.extend(sorted(opt_params, key=lambda p: p.name))
+
+        param_strs = []
+        for p in parameters:
+            one = p.to_typescript_style(indent=indent + _TS_INDENT)
+            param_strs.append(one)
+
+        if self.additional_properties is not None:
+            ap_type_str = "any"
+            if self.additional_properties is True:
+                ap_type_str = "any"
+            elif self.additional_properties is False:
+                ap_type_str = "never"
+            elif isinstance(self.additional_properties, _ParameterType):
+                ap_type_str = self.additional_properties.to_typescript_style(
+                    indent=indent + _TS_INDENT
+                )
+            else:
+                raise ValueError(
+                    f"Unknown additionalProperties: {self.additional_properties}"
+                )
+            param_strs.append(f"{indent + _TS_INDENT}[k: string]: {ap_type_str}")
+
+        if not param_strs:
+            return "{}"
+
+        params_str = _TS_FIELD_DELIMITER.join(param_strs)
+        if params_str:
+            # add new line before and after
+            params_str = f"\n{params_str}\n"
+        # always wrap with object
+        return f"{{{params_str}{indent}}}"
+
+
+class _ParameterTypeArray(_BaseType):
+    item: "_ParameterType"
+
+    def __init__(
+        self,
+        json_schema_object: dict[str, Any],
+        registry: _SchemaRegistry | None = None,
+    ):
+        super().__init__(
+            json_schema_object, allowed_constraint_keys=("minItems", "maxItems")
+        )
+        if json_schema_object.get("items"):
+            self.item = _parse_parameter_type(json_schema_object["items"], registry)
+        else:
+            self.item = _ParameterTypeScalar(type="any")
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        item_docstring = self.item.format_docstring(indent + _TS_INDENT)
+        if item_docstring:
+            return (
+                "Array<\n"
+                + item_docstring
+                + indent
+                + _TS_INDENT
+                + self.item.to_typescript_style(indent=indent + _TS_INDENT)
+                + "\n"
+                + indent
+                + ">"
+            )
+        else:
+            return f"Array<{self.item.to_typescript_style(indent=indent)}>"
+
+
+class _ParameterTypeEnum(_BaseType):
+    # support scalar types only
+    enum: list[str | int | float | bool | None]
+
+    def __init__(self, json_schema_object: dict[str, Any]):
+        super().__init__(json_schema_object)
+        self.enum = json_schema_object["enum"]
+
+        # Validate enum values against declared type if present
+        if "type" in json_schema_object:
+            typ = json_schema_object["type"]
+            if isinstance(typ, list):
+                if len(typ) == 1:
+                    typ = typ[0]
+                elif len(typ) == 2:
+                    if "null" not in typ:
+                        raise ValueError(f"Enum type {typ} is not supported")
+                    else:
+                        typ = typ[0] if typ[0] != "null" else typ[1]
+                else:
+                    raise ValueError(f"Enum type {typ} is not supported")
+            for val in self.enum:
+                if val is None:
+                    continue
+                if typ == "string" and not isinstance(val, str):
+                    raise ValueError(f"Enum value {val} is not a string")
+                elif typ == "number" and not isinstance(val, (int, float)):
+                    raise ValueError(f"Enum value {val} is not a number")
+                elif typ == "integer" and not isinstance(val, int):
+                    raise ValueError(f"Enum value {val} is not an integer")
+                elif typ == "boolean" and not isinstance(val, bool):
+                    raise ValueError(f"Enum value {val} is not a boolean")
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        return " | ".join(
+            [f'"{e}"' if isinstance(e, str) else str(e) for e in self.enum]
+        )
+
+
+class _ParameterTypeAnyOf(_BaseType):
+    types: list["_ParameterType"]
+
+    def __init__(
+        self,
+        json_schema_object: dict[str, Any],
+        registry: _SchemaRegistry | None = None,
+    ):
+        super().__init__(json_schema_object)
+        self.types = [
+            _parse_parameter_type(t, registry) for t in json_schema_object["anyOf"]
+        ]
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        return " | ".join([t.to_typescript_style(indent=indent) for t in self.types])
+
+
+class _ParameterTypeUnion(_BaseType):
+    types: list[str]
+
+    def __init__(self, json_schema_object: dict[str, Any]):
+        super().__init__(json_schema_object)
+
+        mapping = {
+            "string": "string",
+            "number": "number",
+            "integer": "number",
+            "boolean": "boolean",
+            "null": "null",
+            "object": "{}",
+            "array": "Array<any>",
+        }
+        self.types = [mapping[t] for t in json_schema_object["type"]]
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        return " | ".join(self.types)
+
+
+class _ParameterTypeRef(_BaseType):
+    ref_name: str
+    is_self_ref: bool = False
+
+    def __init__(self, json_schema_object: dict[str, Any], registry: _SchemaRegistry):
+        super().__init__(json_schema_object)
+
+        ref = json_schema_object["$ref"]
+        resolved_schema = registry.resolve_ref(ref)
+
+        if resolved_schema.get("$self_ref", False):
+            self.ref_name = "parameters"
+            self.is_self_ref = True
+        else:
+            self.ref_name = ref.split("/")[-1]
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        return self.ref_name
+
+
+_ParameterType = (
+    _ParameterTypeScalar
+    | _ParameterTypeObject
+    | _ParameterTypeArray
+    | _ParameterTypeEnum
+    | _ParameterTypeAnyOf
+    | _ParameterTypeUnion
+    | _ParameterTypeRef
+)
+
+
+@dataclasses.dataclass
+class _Parameter:
+    """
+    A parameter in a function, or a field in a object.
+    It consists of the type as well as the name.
+    """
+
+    type: _ParameterType
+    name: str = "_"
+    optional: bool = True
+    default: Any | None = None
+
+    @classmethod
+    def parse_extended(cls, attributes: dict[str, Any]) -> "_Parameter":
+        if not attributes:
+            raise ValueError("attributes is empty")
+
+        return cls(
+            name=attributes.get("name", "_"),
+            type=_parse_parameter_type(attributes),
+            optional=attributes.get("optional", False),
+            default=attributes.get("default"),
+        )
+
+    def to_typescript_style(self, indent: str = "") -> str:
+        comments = self.type.format_docstring(indent)
+
+        if self.default is not None:
+            default_repr = (
+                json.dumps(self.default, ensure_ascii=False)
+                if not isinstance(self.default, (int, float, bool))
+                else repr(self.default)
+            )
+            comments += f"{indent}// Default: {default_repr}\n"
+
+        return (
+            comments
+            + f"{indent}{self.name}{'?' if self.optional else ''}: {self.type.to_typescript_style(indent=indent)}"
+        )
+
+
+def _parse_parameter_type(
+    json_schema_object: dict[str, Any] | bool, registry: _SchemaRegistry | None = None
+) -> _ParameterType:
+    if isinstance(json_schema_object, bool):
+        if json_schema_object:
+            return _ParameterTypeScalar(type="any")
+        else:
+            logger.warning(
+                f"Warning: Boolean value {json_schema_object} is not supported, use null instead."
+            )
+            return _ParameterTypeScalar(type="null")
+
+    if "$ref" in json_schema_object and registry:
+        return _ParameterTypeRef(json_schema_object, registry)
+
+    if "anyOf" in json_schema_object:
+        return _ParameterTypeAnyOf(json_schema_object, registry)
+    elif "enum" in json_schema_object:
+        return _ParameterTypeEnum(json_schema_object)
+    elif "type" in json_schema_object:
+        typ = json_schema_object["type"]
+        if isinstance(typ, list):
+            return _ParameterTypeUnion(json_schema_object)
+        elif typ == "object":
+            return _ParameterTypeObject(json_schema_object, registry)
+        elif typ == "array":
+            return _ParameterTypeArray(json_schema_object, registry)
+        else:
+            return _ParameterTypeScalar(typ, json_schema_object)
+    elif json_schema_object == {}:
+        return _ParameterTypeScalar(type="any")
+    else:
+        raise ValueError(f"Invalid JSON Schema object: {json_schema_object}")
+
+
+def _openai_function_to_typescript_style(
+    function: dict[str, Any],
+) -> str:
+    """Convert OpenAI function definition (dict) to TypeScript style string."""
+    registry = _SchemaRegistry()
+    parameters = function.get("parameters") or {}
+    parsed = _ParameterTypeObject(parameters, registry)
+
+    interfaces = []
+    root_interface_name = None
+    if registry.has_self_ref:
+        root_interface_name = "parameters"
+        params_str = _TS_FIELD_DELIMITER.join(
+            [p.to_typescript_style(indent=_TS_INDENT) for p in parsed.properties]
+        )
+        params_str = f"\n{params_str}\n" if params_str else ""
+        interface_def = f"interface {root_interface_name} {{{params_str}}}"
+        interfaces.append(interface_def)
+
+    definitions_copy = dict(registry.definitions)
+    for def_name, def_schema in definitions_copy.items():
+        obj_type = _parse_parameter_type(def_schema, registry)
+        params_str = obj_type.to_typescript_style()
+
+        description_part = ""
+        if obj_description := def_schema.get("description", ""):
+            description_part = _format_description(obj_description) + "\n"
+
+        interface_def = f"{description_part}interface {def_name} {params_str}"
+        interfaces.append(interface_def)
+
+    interface_str = "\n".join(interfaces)
+    function_name = function.get("name", "function")
+    if root_interface_name:
+        type_def = f"type {function_name} = (_: {root_interface_name}) => any;"
+    else:
+        params_str = parsed.to_typescript_style()
+        type_def = f"type {function_name} = (_: {params_str}) => any;"
+
+    description = function.get("description")
+    return "\n".join(
+        filter(
+            bool,
+            [
+                interface_str,
+                ((description and _format_description(description)) or ""),
+                type_def,
+            ],
+        )
+    )
+
+
+def encode_tools_to_typescript_style(
+    tools: list[dict[str, Any]],
+) -> str:
+    """
+    Convert tools (list of dict) to TypeScript style string.
+
+    Supports OpenAI format: {"type": "function", "function": {...}}
+
+    Args:
+        tools: List of tool definitions in dict format
+
+    Returns:
+        TypeScript style string representation of the tools
+    """
+    if not tools:
+        return ""
+
+    functions = []
+
+    for tool in tools:
+        tool_type = tool.get("type")
+        if tool_type == "function":
+            func_def = tool.get("function", {})
+            if func_def:
+                functions.append(_openai_function_to_typescript_style(func_def))
+        else:
+            # Skip unsupported tool types (like "_plugin")
+            continue
+
+    if not functions:
+        return ""
+
+    functions_str = "\n".join(functions)
+    result = "# Tools\n\n"
+
+    if functions_str:
+        result += "## functions\nnamespace functions {\n"
+        result += functions_str + "\n"
+        result += "}\n"
+
+    return result