---
license: mit
language:
- ru
- en
pipeline_tag: text-generation
tags:
- dllm
- diffusion
- diffusion-language-modeling
- instruct
library_name: transformers
---
# Zarya-0.6B

Zarya is a family of hybrid language models that combine a classic auto-regressive (AR) objective with a masked-diffusion (MDM) objective in one model.
The architecture can be built on top of any autoregressive model but in this repository it uses the `Qwen3` backbone.

Naming explanation: Zarya (pronounced as [zɐˈrʲa] ([IPA notation](https://en.wiktionary.org/wiki/Appendix:Russian_pronunciation)), literally "Dawn" in English) is a figure from Slavic folklore — a female personification of dawn who may be considered a goddess.
In various traditions, she can manifest as a single being or as two or three sisters simultaneously.

This is a research prototype.

## Model Details

### Model Description

Zarya is a research prototype of a family of hybrid language models that jointly learn a classic auto-regressive (AR) objective and a masked-diffusion (MDM) objective within a single model.

Two generation modes are supported, both reachable through a single `model.generate(...)` call: masked-diffusion (MDM) sampling and slotted-level speculative parallel decoding.

- **Model type:** Hybrid auto-regressive (AR) + masked-diffusion language model (DLLM); backbone `Qwen3`, wrapper `Zarya`
- **Language(s) (NLP):** Russian and English
- **License:** MIT
- **Preprint:** https://arxiv.org/abs/2609.19868
- **Repository with training code:** https://github.com/ai-forever/zarya

### Zarya-0.6B details
Zarya-0.6B has the following features:

| Variant    | hidden_size | num_hidden_layers | num_attention_heads | intermediate_size |
|------------|-------------|-------------------|---------------------|-------------------|
| Zarya-0.6B | 1024        | 28                | 16                  | 3072              |

Context Length: 2048

## Uses

Zarya is intended for text generation.
It supports conversational fine-tuning (SFT) and classic auto-regressive pretraining.

### Direct Use

Direct use is text generation (continuation of a prompt) through the `model.generate(...)` interface, including chat-style prompts formatted with the provided chat template.
Two inference modes are available through the same `generate()` call.
Both modes fully use the KV cache with causal attention masks.
- **MDM sampling** (`slotted_generation=false`): iterative masked-diffusion denoising with the first-hitting sampler.
- **Slotted speculative decoding** (`slotted_generation=true`): parallel slot generation with inter-slot diffusion-based selection and intra-slot autoregressive generation for a decoding speedup.

### Out-of-Scope Use

The model is a research prototype.
It should not be used for production decisions, safety-critical applications, or any use case where accuracy and reliability are essential without additional evaluation and safeguards.
Inference performance and stability also depend on the chosen decoding hyperparameters (like `slotted_generation`, `slot_size`, `serial_num_blocks`, `slot_threshold`, `token_threshold`, and others).

## Bias, Risks, and Limitations

This is a research prototype.
The code relies on Hugging Face Transformers APIs; when upgrading versions, compatibility must be checked (tested on Transformers 5.12.1 and PyTorch 2.9.0).
Inference performance and stability depend on the choice of config parameters.

## How to Get Started with the Model

Use the code below to get started with the model. Loading the model and tokenizer requires `trust_remote_code=True`.

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_name = "ai-forever/Zarya-0.6B"

model = AutoModel.from_pretrained(model_name, trust_remote_code=True, torch_dtype=torch.bfloat16).cuda()
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

prompt = "<|im_start|>user\nHello!<|im_end|>\n<|im_start|>assistant\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()

# Both modes go through model.generate(...).
# With generation_config.slotted_generation=true -> slotted speculative decoding:
out = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    slot_size=16,
    serial_num_blocks=4,
    slot_threshold=0.9,
    token_threshold=0.3,
)
# Setting generation_config.slotted_generation=false -> MDM sampling instead:
# out = model.generate(input_ids, max_new_tokens=256)

print(tokenizer.decode(out[0]))
```


## Evaluation

LM-eval benchmarking with the `lm-eval` package is supported. Example run:

```bash
lm_eval run \
--tasks=gsm8k,ifeval,arc_challenge,hendrycks_math500,humaneval_instruct,humaneval,mbpp,mbpp_plus,hellaswag \
--model=hf --confirm_run_unsafe_code \
--log_samples \
--apply_chat_template \
--output_path=./reports/lm-eval_results \
--model_args=pretrained=ai-forever/Zarya-0.6B,backend=causal,dtype=bfloat16,attn_implementation=sdpa,trust_remote_code=True \
--gen_kwargs slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4
```

### Zarya-0.6B results

Measurements below were collected with varying inference parameters and on different GPUs; performance is sensitive to both, so results may differ across configurations and hardware setups.

#### A100, `dtype=bfloat16`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`

Hardware info:
gpu_driver_cuda_version 13.2;
gpu_driver_version 595.71.05;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks     | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|-----------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| gsm8k     |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.2646 | ± | 0.0122 |
|           |         | strict-match     |      5 | exact_match             | ↑ | 0.2646 | ± | 0.0122 |
| hellaswag |       1 | none             |      0 | acc                     | ↑ | 0.3526 | ± | 0.0048 |
|           |         | none             |      0 | acc_norm                | ↑ | 0.4270 | ± | 0.0049 |
| ifeval    |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.5372 | ± | N/A    |
|           |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.5108 | ± | N/A    |
|           |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.4177 | ± | 0.0212 |
|           |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.3993 | ± | 0.0211 |
| mbpp      |       1 | none             |      3 | pass_at_1               | ↑ | 0.1500 | ± | 0.0160 |
| mbpp_plus |       1 | none             |      3 | pass_at_1               | ↑ | 0.2249 | ± | 0.0215 |


#### H100, `dtype=bfloat16`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.105.08;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3063 | ± | 0.0135 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3464 | ± | 0.0139 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0379 | ± | 0.0053 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0243 | ± | 0.0042 |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4266 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0280 | ± | 0.0074 |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0671 | ± | 0.0196 |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.4365 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.4161 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2884 | ± | 0.0195 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2662 | ± | 0.0190 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |

#### H100, `dtype=bfloat16`, `apply_chat_template`, `slotted_generation=false,T=0,temperature=0.5,top_p=0.8,do_sample=true,noise_schedule=linear`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.105.08;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3063 | ± | 0.0135 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3464 | ± | 0.0139 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0091 | ± | 0.0026 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0000 | ± | 0      |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4266 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.2230 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.1906 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.1257 | ± | 0.0143 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.1035 | ± | 0.0131 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |


#### H100, `dtype=bfloat16`, `slotted_generation=false,T=0,temperature=0.5,top_p=0.8,do_sample=true,noise_schedule=linear`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.126.20;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.2850 | ± | 0.0132 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3046 | ± | 0.0134 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0114 | ± | 0.0029 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0015 | ± | 0.0011 |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3295 | ± | 0.0047 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4018 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.1942 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.1894 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.0980 | ± | 0.0128 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.0943 | ± | 0.0126 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |

#### H100, `dtype=bfloat16`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.105.08;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.2850 | ± | 0.0132 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3046 | ± | 0.0134 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0129 | ± | 0.0031 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0250 | ± | 0.0043 |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3295 | ± | 0.0047 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4018 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0305 | ± | 0.0135 |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0122 | ± | 0.0086 |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.3921 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.3489 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2625 | ± | 0.0189 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2274 | ± | 0.0180 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |

#### H100, `dtype=float32`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.126.20;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3055 | ± | 0.0135 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3456 | ± | 0.0139 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0250 | ± | 0.0043 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0136 | ± | 0.0032 |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4281 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0200 | ± | 0.0063 |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0671 | ± | 0.0196 |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.3993 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.3849 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2754 | ± | 0.0192 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2643 | ± | 0.0190 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |


#### H100, `dtype=float32`, `apply_chat_template`, `slotted_generation=false,T=0,temperature=0.5,top_p=0.8,do_sample=true,noise_schedule=linear`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.126.20;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3055 | ± | 0.0135 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3456 | ± | 0.0139 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0121 | ± | 0.0030 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0008 | ± | 0.0008 |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4281 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.2362 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.1966 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.1423 | ± | 0.0150 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.1091 | ± | 0.0134 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |


#### H100, `dtype=float16`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.126.20;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3055 | ± | 0.0135 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3447 | ± | 0.0139 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0235 | ± | 0.0042 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0159 | ± | 0.0034 |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4277 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0180 | ± | 0.0060 |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0732 | ± | 0.0204 |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.4077 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.3981 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2754 | ± | 0.0192 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2606 | ± | 0.0189 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |


#### H100, `dtype=float16`, `apply_chat_template`, `slotted_generation=false,T=0,temperature=0.5,top_p=0.8,do_sample=true,noise_schedule=linear`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.126.20;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3055 | ± | 0.0135 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3447 | ± | 0.0139 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0114 | ± | 0.0029 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0000 | ± | 0      |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4277 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0000 | ± | 0      |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.2410 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.2014 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.1460 | ± | 0.0152 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.1128 | ± | 0.0136 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |


#### H100, `dtype=bfloat16`, `apply_chat_template`, `slotted_generation=true,slot_size=16,serial_num_blocks=4,slot_threshold=0.9,token_threshold=0.4,max_gen_toks=2048`

Hardware info:
gpu_driver_cuda_version 13.0;
gpu_driver_version 580.105.08;

Docker info:
Torch: 2.9.0+cu128; Transformers: 5.12.1; CUDNN in torch: 91002;
`lm-eval == 0.4.12`

| Tasks              | Version | Filter           | n-shot | Metric                  |   |  Value |   | Stderr |
|--------------------|--------:|------------------|-------:|-------------------------|---|-------:|---|--------|
| arc_challenge      |       1 | none             |      0 | acc                     | ↑ | 0.3063 | ± | 0.0135 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.3464 | ± | 0.0139 |
| gsm8k              |       3 | flexible-extract |      5 | exact_match             | ↑ | 0.0182 | ± | 0.0037 |
|                    |         | strict-match     |      5 | exact_match             | ↑ | 0.0023 | ± | 0.0013 |
| hellaswag          |       1 | none             |      0 | acc                     | ↑ | 0.3525 | ± | 0.0048 |
|                    |         | none             |      0 | acc_norm                | ↑ | 0.4266 | ± | 0.0049 |
| hendrycks_math500  |       1 | none             |      0 | exact_match             | ↑ | 0.0240 | ± | 0.0069 |
| humaneval          |       1 | create_test      |      0 | pass@1                  | ↑ | 0.0000 | ± | 0      |
| humaneval_instruct |       4 | create_test      |      0 | pass@1                  | ↑ | 0.0671 | ± | 0.0196 |
| ifeval             |       4 | none             |      0 | inst_level_loose_acc    | ↑ | 0.3933 | ± | N/A    |
|                    |         | none             |      0 | inst_level_strict_acc   | ↑ | 0.3789 | ± | N/A    |
|                    |         | none             |      0 | prompt_level_loose_acc  | ↑ | 0.2680 | ± | 0.0191 |
|                    |         | none             |      0 | prompt_level_strict_acc | ↑ | 0.2606 | ± | 0.0189 |
| mbpp               |       1 | none             |      3 | pass_at_1               | ↑ | 0.0020 | ± | 0.0020 |
| mbpp_plus          |       1 | none             |      3 | pass_at_1               | ↑ | 0.0000 | ± | 0      |

---

## Citation

If you find our work helpful, please consider citing (citation will be updated after peer-reviewed publication):

```bibtex
@misc{sinev-etal-2026-Zarya,
  author        = {Sinev, Leonid and Koziev, Ilya and Leshchuk, Vladislav},
  title         = {Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference},
  year          = {2026},
  archiveprefix = {arXiv},
  eprint        = {2609.19868},
  primaryclass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.19868},
}
```