Zarya-0.6B model initial upload

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,423 @@
+---
+license: mit
+language:
+- ru
+- en
+pipeline_tag: text-generation
+tags:
+- dllm
+- diffusion
+- diffusion-language-modeling
+- instruct
+library_name: transformers
+---
+# Zarya-0.6B
+
+Zarya is a family of hybrid language models that combine a classic auto-regressive (AR) objective with a masked-diffusion (MDM) objective in one model.
+The architecture can be built on top of any autoregressive model but in this repository it uses the `Qwen3` backbone.
+
+Naming explanation: Zarya (pronounced as [zɐˈrʲa] ([IPA notation](https://en.wiktionary.org/wiki/Appendix:Russian_pronunciation)), literally "Dawn" in English) is a figure from Slavic folklore — a female personification of dawn who may be considered a goddess.
+In various traditions, she can manifest as a single being or as two or three sisters simultaneously.
+
+This is a research prototype.
+
+## Model Details
+
+### Model Description
+
+Zarya is a research prototype of a family of hybrid language models that jointly learn a classic auto-regressive (AR) objective and a masked-diffusion (MDM) objective within a single model.
+
+Two generation modes are supported, both reachable through a single `model.generate(...)` call: masked-diffusion (MDM) sampling and slotted-level speculative parallel decoding.
+
+- **Model type:** Hybrid auto-regressive (AR) + masked-diffusion language model (DLLM); backbone `Qwen3`, wrapper `Zarya`
+- **Language(s) (NLP):** Russian and English
+- **License:** MIT
+- **Preprint:** https://arxiv.org/abs/2609.19868
+- **Repository with training code:** https://github.com/ai-forever/zarya
+
+### Zarya-0.6B details
+Zarya-0.6B has the following features:
+
+| Variant    | hidden_size | num_hidden_layers | num_attention_heads | intermediate_size |
+|------------|-------------|-------------------|---------------------|-------------------|
+| Zarya-0.6B | 1024        | 28                | 16                  | 3072              |
+
+Context Length: 2048
+
+## Uses
+
+Zarya is intended for text generation.
+It supports conversational fine-tuning (SFT) and classic auto-regressive pretraining.
+
+### Direct Use
+
+Direct use is text generation (continuation of a prompt) through the `model.generate(...)` interface, including chat-style prompts formatted with the provided chat template.
+Two inference modes are available through the same `generate()` call.
+Both modes fully use the KV cache with causal attention masks.
+- **MDM sampling** (`slotted_generation=false`): iterative masked-diffusion denoising with the first-hitting sampler.
+- **Slotted speculative decoding** (`slotted_generation=true`): parallel slot generation with inter-slot diffusion-based selection and intra-slot autoregressive generation for a decoding speedup.
+
+### Out-of-Scope Use
+
+The model is a research prototype.
+It should not be used for production decisions, safety-critical applications, or any use case where accuracy and reliability are essential without additional evaluation and safeguards.
+Inference performance and stability also depend on the chosen decoding hyperparameters (like `slotted_generation`, `slot_size`, `serial_num_blocks`, `slot_threshold`, `token_threshold`, and others).
+
+## Bias, Risks, and Limitations
+
+This is a research prototype.
+The code relies on Hugging Face Transformers APIs; when upgrading versions, compatibility must be checked (tested on Transformers 5.12.1 and PyTorch 2.9.0).
+Inference performance and stability depend on the choice of config parameters.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model. Loading the model and tokenizer requires `trust_remote_code=True`.
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+model_name = "ai-forever/Zarya-0.6B"
+
+model = AutoModel.from_pretrained(model_name, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+
+prompt = "<|im_start|>user\nHello!<|im_end|>\n<|im_start|>assistant\n"
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
+
+# Both modes go through model.generate(...).
+# With generation_config.slotted_generation=true -> slotted speculative decoding:
+out = model.generate(
+    input_ids,
+    max_new_tokens=256,
+    do_sample=True,
+    temperature=0.7,
+    slot_size=16,
+    serial_num_blocks=4,
+    slot_threshold=0.9,
+    token_threshold=0.3,
+)
+# Setting generation_config.slotted_generation=false -> MDM sampling instead:
+# out = model.generate(input_ids, max_new_tokens=256)
+
+print(tokenizer.decode(out[0]))
+```
+
+
+## Evaluation
+
+LM-eval benchmarking with the `lm-eval` package is supported. Example run:
+
+```bash
+lm_eval run \
+--tasks=gsm8k,ifeval,arc_challenge,hendrycks_math500,humaneval_instruct,humaneval,mbpp,mbpp_plus,hellaswag \
+--model=hf --confirm_run_unsafe_code \
+--log_samples \
+--apply_chat_template \
+--output_path=./reports/lm-eval_results \
+--model_args=pretrained=ai-forever/Zarya-0.6B,backend=causal,dtype=bfloat16,attn_implementation=sdpa,trust_remote_code=True \
+--gen_kwargs slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4
+```
+
+### Zarya-0.6B results
+
+Measurements below were collected with varying inference parameters and on different GPUs; performance is sensitive to both, so results may differ across configurations and hardware setups.
+
+#### A100, `dtype=bfloat16`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`
+
+Hardware info:
+gpu_driver_cuda_version 13.2;
+gpu_driver_version 595.71.05;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks     | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|-----------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| gsm8k     |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.2646 | ± | 0.0122 |
+|           |         | strict-match     |      5 | exact_match             | ↑ | 0.2646 | ± | 0.0122 |
+| hellaswag |       1 | none             |      0 | acc                     | ↑ | 0.3526 | ± | 0.0048 |
+|           |         | none             |      0 | acc_norm                | ↑ | 0.4270 | ± | 0.0049 |
+| ifeval    |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.5372 | ± | N/A    |
+|           |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.5108 | ± | N/A    |
+|           |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.4177 | ± | 0.0212 |
+|           |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.3993 | ± | 0.0211 |
+| mbpp      |       1 | none             |      3 | pass_at_1               | ↑ | 0.1500 | ± | 0.0160 |
+| mbpp_plus |       1 | none             |      3 | pass_at_1               | ↑ | 0.2249 | ± | 0.0215 |
+
+
+#### H100, `dtype=bfloat16`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.105.08;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3063 | ± | 0.0135 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3464 | ± | 0.0139 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0379 | ± | 0.0053 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0243 | ± | 0.0042 |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4266 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0280 | ± | 0.0074 |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0671 | ± | 0.0196 |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.4365 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.4161 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2884 | ± | 0.0195 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2662 | ± | 0.0190 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+#### H100, `dtype=bfloat16`, `apply_chat_template`, `slotted_generation=false,T=0,temperature=0.5,top_p=0.8,do_sample=true,noise_schedule=linear`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.105.08;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3063 | ± | 0.0135 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3464 | ± | 0.0139 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0091 | ± | 0.0026 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0000 | ± | 0      |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4266 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.2230 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.1906 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.1257 | ± | 0.0143 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.1035 | ± | 0.0131 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+
+#### H100, `dtype=bfloat16`, `slotted_generation=false,T=0,temperature=0.5,top_p=0.8,do_sample=true,noise_schedule=linear`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.126.20;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.2850 | ± | 0.0132 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3046 | ± | 0.0134 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0114 | ± | 0.0029 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0015 | ± | 0.0011 |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3295 | ± | 0.0047 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4018 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.1942 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.1894 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.0980 | ± | 0.0128 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.0943 | ± | 0.0126 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+#### H100, `dtype=bfloat16`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.105.08;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.2850 | ± | 0.0132 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3046 | ± | 0.0134 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0129 | ± | 0.0031 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0250 | ± | 0.0043 |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3295 | ± | 0.0047 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4018 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0305 | ± | 0.0135 |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0122 | ± | 0.0086 |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.3921 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.3489 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2625 | ± | 0.0189 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2274 | ± | 0.0180 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+#### H100, `dtype=float32`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.126.20;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3055 | ± | 0.0135 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3456 | ± | 0.0139 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0250 | ± | 0.0043 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0136 | ± | 0.0032 |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4281 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0200 | ± | 0.0063 |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0671 | ± | 0.0196 |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.3993 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.3849 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2754 | ± | 0.0192 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2643 | ± | 0.0190 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+
+#### H100, `dtype=float32`, `apply_chat_template`, `slotted_generation=false,T=0,temperature=0.5,top_p=0.8,do_sample=true,noise_schedule=linear`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.126.20;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3055 | ± | 0.0135 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3456 | ± | 0.0139 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0121 | ± | 0.0030 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0008 | ± | 0.0008 |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4281 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.2362 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.1966 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.1423 | ± | 0.0150 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.1091 | ± | 0.0134 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+
+#### H100, `dtype=float16`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.126.20;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3055 | ± | 0.0135 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3447 | ± | 0.0139 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0235 | ± | 0.0042 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0159 | ± | 0.0034 |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4277 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0180 | ± | 0.0060 |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0732 | ± | 0.0204 |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.4077 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.3981 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2754 | ± | 0.0192 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2606 | ± | 0.0189 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+
+#### H100, `dtype=float16`, `apply_chat_template`, `slotted_generation=false,T=0,temperature=0.5,top_p=0.8,do_sample=true,noise_schedule=linear`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.126.20;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3055 | ± | 0.0135 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3447 | ± | 0.0139 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0114 | ± | 0.0029 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0000 | ± | 0      |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4277 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.2410 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.2014 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.1460 | ± | 0.0152 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.1128 | ± | 0.0136 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+
+#### H100, `dtype=bfloat16`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4,max_gen_toks=2048`
+
+Hardware info:
+gpu_driver_cuda_version 13.0;
+gpu_driver_version 580.105.08;
+
+Docker info:
+Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
+`lm-eval == 0.4.12`
+
+| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
+|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
+| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3063 | ± | 0.0135 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.3464 | ± | 0.0139 |
+| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0182 | ± | 0.0037 |
+|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0023 | ± | 0.0013 |
+| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
+|                    |         | none             |      0 | acc_norm                | ↑ | 0.4266 | ± | 0.0049 |
+| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0240 | ± | 0.0069 |
+| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
+| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0671 | ± | 0.0196 |
+| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.3933 | ± | N/A    |
+|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.3789 | ± | N/A    |
+|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2680 | ± | 0.0191 |
+|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2606 | ± | 0.0189 |
+| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0020 | ± | 0.0020 |
+| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
+
+---
+
+## Citation
+
+If you find our work helpful, please consider citing (citation will be updated after peer-reviewed publication):
+
+```bibtex
+@misc{sinev-etal-2026-Zarya,
+  author        = {Sinev, Leonid and Koziev, Ilya and Leshchuk, Vladislav},
+  title         = {Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference},
+  year          = {2026},
+  archiveprefix = {arXiv},
+  eprint        = {2609.19868},
+  primaryclass  = {cs.CL},
+  url           = {https://arxiv.org/abs/2609.19868},
+}
+```

chat_template.jinja

@@ -0,0 +1,89 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {{- messages[0].content + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if loop.last or (not loop.last and reasoning_content) %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,91 @@
+{
+  "T": 0,
+  "add_loss_path": false,
+  "alpha_0": 0.75,
+  "architectures": [
+    "Zarya"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration.ZaryaConfig",
+    "AutoModel": "modeling.Zarya",
+    "AutoModelForCausalLM": "modeling.Zarya",
+    "AutoModelForMaskedLM": "modeling.Zarya"
+  },
+  "bos_token_id": 151644,
+  "diffusion_attn_mode": "causal",
+  "diffusion_loss_proportion": 0.5,
+  "diffusion_shuffle": true,
+  "dropout": 0.1,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "extra_processing": true,
+  "grouped_noise": false,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "mask_token_id": 151669,
+  "max_position_embeddings": 40960,
+  "max_window_layers": 28,
+  "model_type": "zarya",
+  "noise_eps": 0.001,
+  "noise_sorting": true,
+  "norm_elementwise_affine": true,
+  "norm_eps": 1e-06,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "ordered_sampling": false,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sample_t_override": 0.0,
+  "sample_t_upper": 0.1,
+  "sampling_eps": 0.001,
+  "scale_by_batch": false,
+  "sequential_attn_mode": "causal",
+  "sequential_shuffle": true,
+  "simple_masking": false,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "time_conditioning": false,
+  "transformers_version": "5.12.1",
+  "unnormalized_loss": false,
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

configuration.py

@@ -0,0 +1,131 @@
+from typing import Optional, Union
+
+from transformers import AutoConfig, AutoModel  # noqa: F401
+from transformers.models.qwen3.configuration_qwen3 import Qwen3Config  # noqa: F401
+
+try:
+    from transformers import PreTrainedConfig  # noqa: F401
+except ImportError:
+    from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig  # noqa: F401
+
+try:
+    from transformers.configuration_utils import layer_type_validation
+except ImportError:
+    layer_type_validation = None
+
+try:
+    from transformers.modeling_rope_utils import RopeParameters
+except ImportError:
+    RopeParameters = None
+
+try:
+    from transformers.modeling_rope_utils import rope_config_validation
+except ImportError:
+    rope_config_validation = None
+
+
+class ZaryaConfig(Qwen3Config):
+    """Configuration class for Zarya model."""
+
+    model_type = "zarya"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # Default tensor parallel plan for base model
+    base_model_tp_plan = {
+        "layers.*.self_attn.q_proj": "colwise",
+        "layers.*.self_attn.k_proj": "colwise",
+        "layers.*.self_attn.v_proj": "colwise",
+        "layers.*.self_attn.q_norm": "replicated_with_grad_allreduce",
+        "layers.*.self_attn.k_norm": "replicated_with_grad_allreduce",
+        "layers.*.self_attn.o_proj": "rowwise",
+        "layers.*.mlp.gate_proj": "colwise",
+        "layers.*.mlp.up_proj": "colwise",
+        "layers.*.mlp.down_proj": "rowwise",
+    }
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+    backbone_class = "Qwen3ForCausalLM"
+    vocab_size: int = 151936
+    hidden_size: int = 1024
+    intermediate_size: int = 22016
+    num_hidden_layers: int = 12
+    num_attention_heads: int = 12
+    num_key_value_heads: Optional[int] = 12
+    head_dim: int = 128
+    hidden_act: str = "silu"
+    max_position_embeddings: int = 2048
+    initializer_range: float = 0.02
+    rms_norm_eps: float = 1e-6
+    use_cache: bool = True
+    tie_word_embeddings: bool = False
+    attention_bias: bool = False
+    use_sliding_window: bool = False
+    sliding_window: Optional[int] = None
+    max_window_layers: int = 28
+    layer_types: Optional[list[str]] = None
+    attention_dropout: Union[float, int] = 0.0
+    pad_token_id: Optional[int] = None
+    bos_token_id: Optional[int] = None
+    eos_token_id: Optional[Union[int, list[int]]] = None
+    dropout: float = 0.1
+    alpha_0: float = 0.25
+    noise_eps: float = 1e-3
+    diffusion_loss_proportion: float = 0.5
+    sequential_attn_mode: str = "mixed"
+    diffusion_attn_mode: str = "mixed"
+    sequential_shuffle: bool = False
+    diffusion_shuffle: bool = False
+    sampling_eps: float = 1e-3
+    time_conditioning: bool = False
+    norm_elementwise_affine: bool = True
+    norm_eps: float = 1e-6
+    T: int = 0
+    slotted_training: bool = True
+    ordered_sampling: bool = False
+    noise_sorting: bool = True
+    scale_by_batch: bool = False
+    unnormalized_loss: bool = False
+    simple_masking: bool = False
+    extra_processing: bool = False
+    sample_t_override: float = 0.0
+    sample_t_upper: float = 1.0
+    add_loss_path: bool = False
+    grouped_noise: bool = False
+    max_span_length: int = 50
+    if RopeParameters is not None:
+        rope_parameters: Optional[Union[RopeParameters, dict]] = None
+    else:
+        rope_theta: Optional[float] = 10000.0
+        rope_scaling: Optional[dict] = None
+
+    def __post_init__(self, **kwargs):
+        self.sliding_window = self.sliding_window if self.use_sliding_window else None
+        if self.num_key_value_heads is None:
+            self.num_key_value_heads = self.num_attention_heads
+
+        if self.layer_types is None:
+            self.layer_types = [
+                "sliding_attention"
+                if self.sliding_window is not None and i >= self.max_window_layers
+                else "full_attention"
+                for i in range(self.num_hidden_layers)
+            ]
+        super().__post_init__(**kwargs)
+
+    def update_from_string(self, update_str: str):
+        super().update_from_string(update_str)
+        if self.layer_types is not None and len(self.layer_types) != self.num_hidden_layers:
+            self.layer_types = [
+                "sliding_attention"
+                if self.sliding_window is not None and i >= self.max_window_layers
+                else "full_attention"
+                for i in range(self.num_hidden_layers)
+            ]
+
+
+ZaryaConfig.register_for_auto_class("AutoConfig")
+AutoConfig.register(ZaryaConfig.model_type, ZaryaConfig)
+__all__ = ["ZaryaConfig"]

generation_config.json

@@ -0,0 +1,21 @@
+{
+  "T": 16,
+  "_from_model_config": false,
+  "adapter_kwargs": {},
+  "bos_token_id": 151644,
+  "diffusion_phase_only": false,
+  "do_sample": false,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "from_tf": false,
+  "ignore_noise_schedule": false,
+  "pad_token_id": 151643,
+  "sequential_phase_only": false,
+  "slotted_generation": true,
+  "transformers_version": "5.12.1",
+  "trust_remote_code": true,
+  "unmask_probs_coef": 1,
+  "use_float64": false
+}

generation_utils.py

@@ -0,0 +1,28 @@
+from transformers.generation.configuration_utils import GenerationConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class ZaryaGenerationConfig(GenerationConfig):
+    model_type = "zarya"
+    ignore_noise_schedule: bool = False
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.ignore_noise_schedule: bool = kwargs.pop("ignore_noise_schedule", False)
+        self.T: int = kwargs.pop("T", 1000)
+        self.use_float64: bool = kwargs.pop("use_float64", False)
+        self.sequential_phase_only: bool = kwargs.pop("sequential_phase_only", False)
+        self.diffusion_phase_only: bool = kwargs.pop("diffusion_phase_only", False)
+        self.unmask_probs_coef: float = kwargs.pop("unmask_probs_coef", 1)
+        self.slot_size: int = kwargs.pop("slot_size", 16)
+        self.serial_num_blocks: int = kwargs.pop("serial_num_blocks", 1)
+        self.slot_threshold: float = kwargs.pop("slot_threshold", 0.9)
+        self.token_threshold: float = kwargs.pop("token_threshold", 0.3)
+
+        # Validate the values of the attributes
+        self.validate(strict=True)
+
+
+__all__ = ["ZaryaGenerationConfig"]

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cfad56b84f3aee103815df5e7a45a0292b83418628f9cc8e3a6eff00611b060
+size 1192137888

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c8165c070bd7b1eff751f9cd82acf52c4636e997f2df755d01f4eac442a3108
+size 11422843

tokenizer_config.json

@@ -0,0 +1,251 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|mdm_mask|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<|mdm_mask|>"
+  ],
+  "bos_token": "<|im_start|>",
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {{- messages[0].content + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}\n        {%- set ns.multi_step_tool = false %}\n        {%- set ns.last_query_index = index %}\n    {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n    {%- if message.content is string %}\n        {%- set content = message.content %}\n    {%- else %}\n        {%- set content = '' %}\n    {%- endif %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n        {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is string %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in content %}\n                {%- set reasoning_content = content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n                {%- set content = content.split('</think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {%- if loop.last or (not loop.last and reasoning_content) %}\n                {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content.strip('\\n') + '\\n</think>\\n\\n' + content.lstrip('\\n') }}\n            {%- else %}\n                {{- '<|im_start|>' + message.role + '\\n' + content }}\n            {%- endif %}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<|mdm_mask|>",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}