Instructions to use yandex/AliceAI-Foundation-80B-A3B-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use yandex/AliceAI-Foundation-80B-A3B-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="yandex/AliceAI-Foundation-80B-A3B-Base", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("yandex/AliceAI-Foundation-80B-A3B-Base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use yandex/AliceAI-Foundation-80B-A3B-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "yandex/AliceAI-Foundation-80B-A3B-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yandex/AliceAI-Foundation-80B-A3B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/yandex/AliceAI-Foundation-80B-A3B-Base

SGLang

How to use yandex/AliceAI-Foundation-80B-A3B-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "yandex/AliceAI-Foundation-80B-A3B-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yandex/AliceAI-Foundation-80B-A3B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "yandex/AliceAI-Foundation-80B-A3B-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "yandex/AliceAI-Foundation-80B-A3B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use yandex/AliceAI-Foundation-80B-A3B-Base with Docker Model Runner:
```
docker model run hf.co/yandex/AliceAI-Foundation-80B-A3B-Base
```

fdrose

timka-byzov

edgarshmavonyan

MaximNikitin08

IrinaLialikova commited on 13 days ago

Commit

aeac2c6

0 Parent(s):

Initial release

Browse files

Co-authored-by: timka-byzov <timka-byzov@users.noreply.huggingface.co>
Co-authored-by: edgarshmavonyan <edgarshmavonyan@users.noreply.huggingface.co>
Co-authored-by: MaximNikitin08 <MaximNikitin08@users.noreply.huggingface.co>
Co-authored-by: IrinaLialikova <IrinaLialikova@users.noreply.huggingface.co>

This view is limited to 50 files because it contains too many changes. See raw diff

Files changed (50) hide show

.gitattributes +36 -0
CONTRIBUTING.md +31 -0
LICENSE +13 -0
NOTICES +8 -0
README.md +294 -0
README_en.md +292 -0
assets/benchmarks.png +3 -0
config.json +58 -0
configuration_alice_ai.py +109 -0
finetune/chat_template.jinja +94 -0
finetune/finetune_lora.py +274 -0
model-00001-of-00049.safetensors +3 -0
model-00002-of-00049.safetensors +3 -0
model-00003-of-00049.safetensors +3 -0
model-00004-of-00049.safetensors +3 -0
model-00005-of-00049.safetensors +3 -0
model-00006-of-00049.safetensors +3 -0
model-00007-of-00049.safetensors +3 -0
model-00008-of-00049.safetensors +3 -0
model-00009-of-00049.safetensors +3 -0
model-00010-of-00049.safetensors +3 -0
model-00011-of-00049.safetensors +3 -0
model-00012-of-00049.safetensors +3 -0
model-00013-of-00049.safetensors +3 -0
model-00014-of-00049.safetensors +3 -0
model-00015-of-00049.safetensors +3 -0
model-00016-of-00049.safetensors +3 -0
model-00017-of-00049.safetensors +3 -0
model-00018-of-00049.safetensors +3 -0
model-00019-of-00049.safetensors +3 -0
model-00020-of-00049.safetensors +3 -0
model-00021-of-00049.safetensors +3 -0
model-00022-of-00049.safetensors +3 -0
model-00023-of-00049.safetensors +3 -0
model-00024-of-00049.safetensors +3 -0
model-00025-of-00049.safetensors +3 -0
model-00026-of-00049.safetensors +3 -0
model-00027-of-00049.safetensors +3 -0
model-00028-of-00049.safetensors +3 -0
model-00029-of-00049.safetensors +3 -0
model-00030-of-00049.safetensors +3 -0
model-00031-of-00049.safetensors +3 -0
model-00032-of-00049.safetensors +3 -0
model-00033-of-00049.safetensors +3 -0
model-00034-of-00049.safetensors +3 -0
model-00035-of-00049.safetensors +3 -0
model-00036-of-00049.safetensors +3 -0
model-00037-of-00049.safetensors +3 -0
model-00038-of-00049.safetensors +3 -0
model-00039-of-00049.safetensors +3 -0

.gitattributes ADDED Viewed

	@@ -0,0 +1,36 @@

+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+benchmarks.png filter=lfs diff=lfs merge=lfs -text

CONTRIBUTING.md ADDED Viewed

	@@ -0,0 +1,31 @@

+## Notice to external contributors
+### General info
+Hello! In order for us (YANDEX LLC) to accept patches and other contributions from you, you will have to adopt our Contributor License Agreement (the “CLA”). The current version of the CLA you may find here:
+* https://yandex.ru/legal/cla/en/ (in English) and
+* https://yandex.ru/legal/cla/ru/ (in Russian).
+By adopting the CLA, you state the following:
+* You obviously wish and are willingly licensing your contributions to us for our open source projects under the terms of the CLA,
+* You have read the terms and conditions of the CLA and agree with them in full,
+* You are legally able to provide and license your contributions as stated,
+* We may use your contributions for our open source projects and for any other our project too,
+* We rely on your assurances concerning the rights of third parties in relation to your contributions.
+If you agree with these principles, please read and adopt our CLA. By providing us your contributions, you hereby declare that you have read and adopted our CLA, and we may freely merge your contributions with our corresponding open source project and use it in further in accordance with terms and conditions of the CLA.
+### Provide contributions
+If you have adopted terms and conditions of the CLA, you are able to provide your contributions. When you submit your pull request, please add the following information into it:
+I hereby agree to the terms of the CLA available at: [link].
+Replace the bracketed text as follows:
+* [link] is the link at the current version of the CLA (you may add here a link https://yandex.ru/legal/cla/?lang=en (in English) or a link https://yandex.ru/legal/cla/?lang=ru (in Russian).
+It is enough to provide us with such notification once.
+### Other questions
+If you have any questions, please write us at opensource-support@yandex-team.ru.

LICENSE ADDED Viewed

	@@ -0,0 +1,13 @@

+Copyright 2026 YANDEX LLC
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+       http://www.apache.org/licenses/LICENSE-2.0
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

NOTICES ADDED Viewed

	@@ -0,0 +1,8 @@

+-------------------------------------------------------------------------------
+Export control notice
+-------------------------------------------------------------------------------
+It is necessary to comply with the applicable export control laws and
+regulations. We declare that we comply with the applicable export control laws
+and regulations for the published software, and we expect the users of the
+software and the contributors to it to be compliant with them as well.

README.md ADDED Viewed

	@@ -0,0 +1,294 @@

+---
+license: apache-2.0
+language:
+  - ru
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - custom_code
+  - mixture-of-experts
+  - vllm
+---
+# AliceAI-Foundation-80B-A3B-Base
+[English version](./README_en.md)
+AliceAI-Foundation-80B-A3B-Base – базовая языковая модель с гибридной
+архитектурой и MoE-слоями. Модель содержит 80 млрд параметров, из которых для каждого токена активируются 3 млрд, и поддерживает контекст длиной до 262 144 токенов. Модель была обучена полностью с нуля.
+При создании модели мы заново собрали обучающий корпус, выбрали архитектуру и
+гиперпараметры, а также подготовили данные для сложных рассуждений и
+взаимодействия с инструментами. Ключевые решения проверялись в серии отдельных
+обучений с нуля объёмом по 2 трлн токенов каждое.
+В математике, программировании и других задачах на рассуждения модель показывает результаты на уровне более крупных опенсорс-моделей, а особенно сильна в задачах на фактические знания на русском языке. Вместе с весами мы публикуем фактологические бенчмарки
+[WikiWebFacts](https://huggingface.co/datasets/yandex/WikiWebFacts) и
+[HardMultiQA](https://huggingface.co/datasets/yandex/HardMultiQA), ориентированные на
+русскоязычный контекст, и протоколы их оценки.
+<img src="./assets/benchmarks.png" alt="Сравнение по бенчмаркам" width="800">
+## Обзор модели
+- Тип: авторегрессионная языковая модель
+- Этап обучения: предобучение
+- Языковая модель
+  - Количетсво параметров: 80B всего, 3B активных
+  - Рзамер скрытого состояния: 2048
+  - Размер словаря: 129024
+  - Количество слоев: 48
+  - Схема слоёв: 12 × (3 × (KDA → MoE) → 1 × (Gated Attention → MoE))
+  - KDA:
+    - Количество query-голов: 32
+    - Количество KV-голов: 32
+    - Размерность query-головы: 128
+    - Размерность KV-головы: 128
+    - Рамер ядра свертки: 4
+  - Gated Attention:
+    - Количество query-голов: 16
+    - Количество KV-голов: 2
+    - Размерность query-головы: 256
+  - MoE:
+    - Количество экспертов: 512
+    - Top-K: 10 + 1 общий эксперт
+    - Промежуточная размерность эксперта: 512
+  - MTP: 1 слой
+- Длина контекста: 262144
+## Бенчмарки
+<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1000px;margin:0 auto;padding:16px 0">
+<p style="margin:0 0 12px;font-size:13px;line-height:1.5">Названия русскоязычных бенчмарков выделены <span style="color:#27834a;font-weight:600">зелёным</span>, англоязычных — <span style="color:#496fa8;font-weight:600">синим</span>.</p>
+<p style="margin:0 0 14px;font-size:13px;line-height:1.5">Все замеры в этом разделе проведены во внутренней инфраструктуре замеров, инференс в фреймворке vllm с t=0 для всех моделей. Жирным выделен победитель в каждой строчке.</p>
+<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
+<colgroup>
+<col style="width:25%">
+<col span="5" style="width:15%">
+</colgroup>
+<thead><tr><th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #d6a15f;color:#b7791f">Бенчмарк</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">AliceAI-Foundation-80B-A3B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">Qwen3.5-35B-A3B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">GLM-4.5-Air-Base (106B-A12B)</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">Nemotron-3-Super-120B-A12B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">DeepSeek-V4-Flash-Base (284B-A13B)</th></tr></thead>
+<tbody>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Факты</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">WikiWebFacts</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк на знание фактов на русском языке.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>86.5</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">62.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">70.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">72.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">83.2</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">HardMultiQA</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк на знание фактов на русском языке.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>67.9</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">47.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">48.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">54.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">65.4</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">CultCat</summary><div style="padding-top:6px;line-height:1.4">4-shot, бенчмарк на знание культурных фактов. Подробнее — в <a href="https://habr.com/ru/companies/yandex/articles/868282/">статье на Хабре</a>.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>86.5</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.1</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">66.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">80.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">TriviaQA</summary><div style="padding-top:6px;line-height:1.4">5-shot, открытый бенчмарк на знание фактов на английском языке; вместо метрики Exact Match используется LLM-as-a-judge.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">79.0</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">83.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>89.8</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">89.4</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Образовательные бенчмарки</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench Russian</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк образования, собранный на основе запросов к Алисе.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>74.2</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">42.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">39.0</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">44.0</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">67.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench Literature</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк образования, собранный на основе запросов к Алисе.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>73.8</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">69.1</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench History</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк образования, собранный на основе запросов к Алисе.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>82.0</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">65.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">62.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">70.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">76.9</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench English</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк образования, собранный на основе запросов к Алисе.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">76.1</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.7</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">67.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>82.9</strong></td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Экспертные знания</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">ExpertFactsQA Medicine</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк на фактические знания, составленный профильными экспертами.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>63.6</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.0</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">42.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">60.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">ExpertFactsQA Law</summary><div style="padding-top:6px;line-height:1.4">Сложный 5-shot, бенчмарк на фактические знания, составленный профильными экспертами.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>49.6</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">27.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">22.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">24.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">40.5</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Экзамены</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EGE CoT</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк из заданий части А ЕГЭ по различным предметам.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>90.5</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">84.7</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">77.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">84.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">90.3</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">MMLU-Pro CoT</summary><div style="padding-top:6px;line-height:1.4">5-shot, открытый бенчмарк на знания и рассуждения по широкому набору предметов на английском языке.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">66.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">63.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">58.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>69.9</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">66.5</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">SuperGPQA CoT</summary><div style="padding-top:6px;line-height:1.4">5-shot, открытый бенчмарк с вопросами, составленными экспертами из разных научных областей.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">44.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">43.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>46.6</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">46.1</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Математика</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">MATH-500</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк с математическими задачами; используется LLM-as-a-judge и более длинные рассуждения в few-shot-примерах.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>91.1</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">81.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">60.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">84.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">80.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench Math</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк образования, собранный на основе запросов к Алисе</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">79.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>80.0</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">56.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">69.7</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">76.3</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench Math University</summary><div style="padding-top:6px;line-height:1.4">5-shot, бенчмарк образования, собранный на основе запросов к Алисе.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>70.1</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">69.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">67.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">68.6</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Код</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">BigCodeBench 1-shot pass@1</summary><div style="padding-top:6px;line-height:1.4">1-shot, наша реализация BigCodeBench с улучшенными тестами.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">48.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">43.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">44.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">48.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>49.1</strong></td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">LiveCodeBench v5-6 CoT 1-shot pass@1</summary><div style="padding-top:6px;line-height:1.4">1-shot, открытый бенчмарк со сложными задачами по программированию, в которых требуется найти алгоритм и реализовать его в коде.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>50.5</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">22.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">38.1</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Длинный контекст</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">FinQA 128k</summary><div style="padding-top:6px;line-height:1.4">5-shot, длинная модификация опенсорсного FinQA, задачи на аналитику по финансовым отчётам.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>74.1</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">73.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.7</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>74.1</strong></td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">LongMemEval 128k</summary><div style="padding-top:6px;line-height:1.4">5-shot, открытый бенчмарк с задачами на поиск и использование информации из длинной истории диалога.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">64.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">64.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>68.0</strong></td></tr>
+</tbody>
+</table>
+</div>
+<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1000px;margin:0 auto;padding:16px 0">
+<p style="margin:0 0 14px;font-size:13px;line-height:1.5">Все замеры в этом разделе проведены во внутренней инфраструктуре замеров, инференс в фреймворке vllm с t=1 и штрафами за повторы (repetition_penalty=1, presence_penalty=1.5) для всех моделей. Жирным выделен победитель в каждой строчке.</p>
+<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
+<colgroup>
+<col style="width:25%">
+<col span="3" style="width:25%">
+</colgroup>
+<thead><tr><th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #d6a15f;color:#b7791f">Бенчмарк</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">AliceAI-Foundation-80B-A3B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">Qwen3.5-35B-A3B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">Nemotron-3-Super-120B-A12B-Base</th></tr></thead>
+<tbody>
+<tr><td colspan="4" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Сложные рассуждения</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">AIME 2026 pass@32</summary><div style="padding-top:6px;line-height:1.4">0-shot, задачи American Invitational Mathematics Examination.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>96.7</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>96.7</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">90.0</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">HMMT 2026 Feb pass@32</summary><div style="padding-top:6px;line-height:1.4">0-shot, задачи февральского математического турнира Harvard–MIT.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>96.9</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">87.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">66.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">IMO Answerbench pass@8</summary><div style="padding-top:6px;line-height:1.4">0-shot, задачи Международной математической олимпиады.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>88.7</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">84.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">64.5</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">CodeForces CPP pass@8</summary><div style="padding-top:6px;line-height:1.4">0-shot, соревновательные задачи Codeforces на C++.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">68.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>73.7</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">56.6</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">LiveCodeBench v5-6 pass@1</summary><div style="padding-top:6px;line-height:1.4">0-shot, открытый бенчмарк с задачами на поиск и использование информации из длинной истории диалога.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>60.4</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">34.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">LiveCodeBench v5-6 pass@8</summary><div style="padding-top:6px;line-height:1.4">0-shot, открытый бенчмарк с задачами на поиск и использование информации из длинной истории диалога.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>82.9</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">82.1</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.8</td></tr>
+</tbody>
+</table>
+</div>
+## Как использовать
+### Transformers
+Модель можно запустить через Transformers. Референсная версия Transformers — 5.16.1. Для выполнения KDA-слоёв на GPU требуется `flash-linear-attention` с поддержкой KDA:
+```bash
+pip install \
+  transformers==5.16.1 \
+  accelerate==1.14.0 \
+  flash-linear-attention==0.5.0
+```
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model_id = "yandex/AliceAI-Foundation-80B-A3B-Base"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype=torch.bfloat16,
+    device_map="auto",
+)
+prompt = "Есть 256 монет с разным весом, за какое минимальное количество попарных взвешиваний можно найти вторую по весу монету?"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+output_ids = model.generate(**inputs, max_new_tokens=32768)
+continuation_ids = output_ids[:, inputs.input_ids.shape[1] :]
+print(tokenizer.decode(continuation_ids[0], skip_special_tokens=True))
+```
+### vLLM
+Также модель можно запустить через vLLM. Для запуска требуются Docker и NVIDIA Container Toolkit.
+```bash
+docker run --name alice-vllm --pull=always --gpus '"device=0,1,2,3"' --ipc=host \
+  -p 8001:8000 \
+  yamlbrand/alice-ai-vllm:latest \
+  yandex/AliceAI-Foundation-80B-A3B-Base \
+  --tensor-parallel-size 4 \
+  --max-model-len auto \
+  --attention-backend FLASH_ATTN \
+  --attention-config.flash_attn_version=2 \
+  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'
+```
+Повторный запуск после остановки, с сохранённым кешем:
+```bash
+docker start -a alice-vllm
+```
+Чтобы использовать все GPU, замените `--gpus '"device=0,1,2,3"'` на `--gpus all`
+и укажите соответствующее значение размера тензорного параллелизма.
+После запуска сервера отправьте запрос:
+```bash
+curl http://127.0.0.1:8001/v1/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "yandex/AliceAI-Foundation-80B-A3B-Base",
+    "prompt": "Есть 256 монет с разным весом, за какое минимальное количество попарных взвешиваний можно найти вторую по весу монету?",
+    "max_tokens": 32768,
+    "temperature": 0
+  }'
+```
+### Токенизатор
+Токенизатор загружается как `LlamaTokenizer` из файла `tokenizer.model`. Модель
+токенизации использует SentencePiece BPE. Маркеры `[COT_ENABLE]`, `[COT_START]`, `[COT_END]`, а также маркеры инструментов являются обычными атомарными токенами словаря, а не специальными токенами Hugging Face.
+В `tokenizer_config.json` для параметра `legacy` явно установлено значение
+`false`, чтобы сохранить ожидаемую обработку пробелов. Не переопределяйте его
+значением `true`.
+### Как дообучить под свои задачи
+#### Формат данных
+Для подготовки агентских данных, использованных при обучении модели, мы
+использовали стандартный формат OpenAI Messages. В нём траектория задаётся
+последовательностью сообщений с ролями `system`, `user`, `assistant`, `tool` и
+`meta`, а определения доступных инструментов передаются отдельно в поле
+`tools`.
+Перед токенизацией каждая такая траектория рендерилась с помощью
+[`chat_template.jinja`](finetune/chat_template.jinja). Шаблон задаёт префиксы
+ролей, представление reasoning-трейсов, описаний инструментов, вызовов функций
+и результатов их вып��лнения. Именно в таком текстовом представлении эти данные
+использовались при обучении модели.
+Для sft и RL рекомендуем хранить данные в формате OpenAI
+Messages и рендерить их с помощью этого шаблона. Так формат новых данных будет
+совпадать с форматом, который модель видела во время претрейна.
+Мы намеренно не указываем этот шаблон как `chat_template` в
+`tokenizer_config.json`: Alice-AI-Foundation-80B-A3B-Base — базовая модель, поэтому у неё нет
+единственного формата диалога, который должен автоматически применяться при
+инференсе. Приведённый шаблон предназначен именно для подготовки данных к
+дообучению.
+#### Пример LoRA-дообучения
+В репозитории есть минимальный пример PEFT-дообучения
+[`finetune_lora.py`](finetune/finetune_lora.py). Он загружает зафиксированную ревизию
+датасета `tatsu-lab/alpaca`, обучается только на ответах и сохраняет только
+LoRA-адаптер. Для модели такого размера необходим FSDP2, пример ниже рассчитан
+на четыре GPU с 80 GB памяти.
+```bash
+pip install \
+  transformers==5.16.1 \
+  accelerate==1.14.0 \
+  peft==0.20.0 \
+  datasets==5.0.1 \
+  flash-linear-attention==0.5.0
+pip install flash-attn==2.8.1 --no-build-isolation
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+accelerate launch \
+  --use_fsdp \
+  --num_processes 4 \
+  --num_machines 1 \
+  --dynamo_backend no \
+  --mixed_precision no \
+  --fsdp_version 2 \
+  --fsdp_reshard_after_forward true \
+  --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP \
+  --fsdp_transformer_layer_cls_to_wrap AliceAIDecoderLayer \
+  --fsdp_cpu_ram_efficient_loading true \
+  --fsdp_sync_module_states true \
+  --fsdp_state_dict_type SHARDED_STATE_DICT \
+  finetune/finetune_lora.py \
+  --model yandex/AliceAI-Foundation-80B-A3B-Base \
+  --steps 100 \
+  --sequence-length 512 \
+  --output-dir alice-lora
+```
+`--mixed_precision no` здесь не означает FP32-модель: базовые веса загружаются
+в BF16, а LoRA-параметры PEFT хранит в FP32.
+При RAM-efficient загрузке полные веса чекпоинта загружает только rank 0;
+остальные процессы создают модель на meta device и получают свои шарды через
+FSDP2.
+---
+Данная модель является предварительно обученной (pretrained) моделью и предоставляется в исходном виде, без дополнительного этапа post-training / alignment.
+Модель предназначена прежде всего для исследований, экспериментов и дальнейшей доработки. Она не является готовым решением для непосредственного использования в пользовательских продуктах и сервисах.
+Перед использованием модели в production-среде рекомендуется провести собственное тестирование и, исходя из сценария применения, реализовать необходимые этапы дообучения, настройки и контроля поведения модели.
+Пользователь самостоятельно определяет применимость модели для конкретного сценария и несет ответственность за ее интеграцию и использование.

README_en.md ADDED Viewed

	@@ -0,0 +1,292 @@

+---
+license: apache-2.0
+language:
+  - ru
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - custom_code
+  - mixture-of-experts
+  - vllm
+---
+# AliceAI-Foundation-80B-A3B-Base
+[Русская версия](./README.md)
+AliceAI-Foundation-80B-A3B-Base is a base language model with a hybrid
+architecture and MoE layers. The model has 80 billion parameters, of which
+3 billion are activated for each token, and supports a context length of up to
+262,144 tokens. The model was trained entirely from scratch.
+To build the model, we assembled a new training corpus, selected the architecture
+and hyperparameters, and prepared data for complex reasoning and tool use. We
+validated key design decisions through a series of separate training runs from
+scratch, each using 2 trillion tokens.
+On mathematics, coding, and other reasoning tasks, the model performs on par
+with larger open-source models and is particularly strong on Russian factual
+knowledge. Alongside the model weights, we release the factual benchmarks
+[WikiWebFacts](https://huggingface.co/datasets/yandex/WikiWebFacts) and
+[HardMultiQA](https://huggingface.co/datasets/yandex/HardMultiQA), which focus on
+Russian-language contexts, together with their evaluation protocols.
+<img src="./assets/benchmarks.png" alt="Benchmark comparison" width="800">
+## Model Overview
+- Type: autoregressive language model
+- Training stage: pre-training
+- Language model
+  - Number of parameters: 80B total, 3B activated
+  - Hidden size: 2048
+  - Vocabulary size: 129024
+  - Number of layers: 48
+  - Layer layout: 12 × (3 × (KDA → MoE) → 1 × (Gated Attention → MoE))
+  - KDA:
+    - Number of query heads: 32
+    - Number of KV heads: 32
+    - Query head dimension: 128
+    - KV head dimension: 128
+    - Convolution kernel size: 4
+  - Gated Attention:
+    - Number of query heads: 16
+    - Number of KV heads: 2
+    - Query head dimension: 256
+  - MoE:
+    - Number of experts: 512
+    - Top-K: 10 routed + 1 shared expert
+    - Expert intermediate size: 512
+  - MTP: 1 layer
+- Context length: 262144
+## Benchmarks
+<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1000px;margin:0 auto;padding:16px 0">
+<p style="margin:0 0 12px;font-size:13px;line-height:1.5">Russian-language benchmark names are shown in <span style="color:#27834a;font-weight:600">green</span>; English-language benchmark names are shown in <span style="color:#496fa8;font-weight:600">blue</span>.</p>
+<p style="margin:0 0 14px;font-size:13px;line-height:1.5">All results in this section were obtained using our internal evaluation infrastructure, with inference performed in vLLM at t=0 for every model. The best result in each row is shown in bold.</p>
+<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
+<colgroup>
+<col style="width:25%">
+<col span="5" style="width:15%">
+</colgroup>
+<thead><tr><th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #d6a15f;color:#b7791f">Benchmark</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">AliceAI-Foundation-80B-A3B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">Qwen3.5-35B-A3B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">GLM-4.5-Air-Base (106B-A12B)</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">Nemotron-3-Super-120B-A12B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">DeepSeek-V4-Flash-Base (284B-A13B)</th></tr></thead>
+<tbody>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Facts</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">WikiWebFacts</summary><div style="padding-top:6px;line-height:1.4">5-shot benchmark of factual knowledge in Russian.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>86.5</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">62.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">70.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">72.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">83.2</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">HardMultiQA</summary><div style="padding-top:6px;line-height:1.4">5-shot benchmark of factual knowledge in Russian.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>67.9</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">47.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">48.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">54.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">65.4</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">CultCat</summary><div style="padding-top:6px;line-height:1.4">4-shot benchmark of cultural knowledge. Read more in our <a href="https://habr.com/ru/companies/yandex/articles/868282/">article on Habr</a>.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>86.5</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.1</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">66.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">80.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">TriviaQA</summary><div style="padding-top:6px;line-height:1.4">5-shot open benchmark of factual knowledge in English; LLM-as-a-judge is used instead of Exact Match.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">79.0</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">83.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>89.8</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">89.4</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Educational benchmarks</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench Russian</summary><div style="padding-top:6px;line-height:1.4">5-shot education benchmark built from queries submitted to Alice.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>74.2</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">42.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">39.0</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">44.0</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">67.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench Literature</summary><div style="padding-top:6px;line-height:1.4">5-shot education benchmark built from queries submitted to Alice.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>73.8</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">69.1</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench History</summary><div style="padding-top:6px;line-height:1.4">5-shot education benchmark built from queries submitted to Alice.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>82.0</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">65.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">62.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">70.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">76.9</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench English</summary><div style="padding-top:6px;line-height:1.4">5-shot education benchmark built from queries submitted to Alice.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">76.1</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.7</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">67.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>82.9</strong></td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Expert knowledge</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">ExpertFactsQA Medicine</summary><div style="padding-top:6px;line-height:1.4">5-shot factual-knowledge benchmark created by domain experts.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>63.6</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.0</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">42.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">60.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">ExpertFactsQA Law</summary><div style="padding-top:6px;line-height:1.4">Challenging 5-shot factual-knowledge benchmark created by domain experts.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>49.6</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">27.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">22.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">24.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">40.5</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Exams</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EGE CoT</summary><div style="padding-top:6px;line-height:1.4">5-shot benchmark based on multiple-choice Unified State Exam tasks across various subjects.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>90.5</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">84.7</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">77.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">84.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">90.3</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">MMLU-Pro CoT</summary><div style="padding-top:6px;line-height:1.4">5-shot open benchmark of knowledge and reasoning across a broad range of subjects in English.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">66.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">63.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">58.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>69.9</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">66.5</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">SuperGPQA CoT</summary><div style="padding-top:6px;line-height:1.4">5-shot open benchmark containing questions written by experts from different scientific fields.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">44.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">43.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>46.6</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">46.1</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Mathematics</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">MATH-500</summary><div style="padding-top:6px;line-height:1.4">5-shot benchmark of mathematical problems; it uses LLM-as-a-judge and longer reasoning traces in the few-shot examples.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>91.1</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">81.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">60.2</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">84.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">80.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench Math</summary><div style="padding-top:6px;line-height:1.4">5-shot education benchmark built from queries submitted to Alice</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">79.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>80.0</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">56.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">69.7</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">76.3</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#27834a">EduBench Math University</summary><div style="padding-top:6px;line-height:1.4">5-shot education benchmark built from queries submitted to Alice.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>70.1</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">69.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">67.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">68.6</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Coding</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">BigCodeBench 1-shot pass@1</summary><div style="padding-top:6px;line-height:1.4">1-shot, our implementation of BigCodeBench with improved tests.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">48.3</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">43.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">44.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">48.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>49.1</strong></td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">LiveCodeBench v5-6 CoT 1-shot pass@1</summary><div style="padding-top:6px;line-height:1.4">1-shot open benchmark of challenging programming problems that require finding an algorithm and implementing it in code.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>50.5</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">22.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.4</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">38.1</td></tr>
+<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Long context</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">FinQA 128k</summary><div style="padding-top:6px;line-height:1.4">5-shot long-context adaptation of the open-source FinQA benchmark, featuring financial-report analysis tasks.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>74.1</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">73.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">35.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">71.7</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>74.1</strong></td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">LongMemEval 128k</summary><div style="padding-top:6px;line-height:1.4">5-shot open benchmark of finding and using information from long dialogue histories.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">64.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">55.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">50.6</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">64.8</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>68.0</strong></td></tr>
+</tbody>
+</table>
+</div>
+<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1000px;margin:0 auto;padding:16px 0">
+<p style="margin:0 0 14px;font-size:13px;line-height:1.5">All results in this section were obtained using our internal evaluation infrastructure, with inference performed in vLLM at t=1 and repetition penalties (repetition_penalty=1, presence_penalty=1.5) for every model. The best result in each row is shown in bold.</p>
+<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
+<colgroup>
+<col style="width:25%">
+<col span="3" style="width:25%">
+</colgroup>
+<thead><tr><th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #d6a15f;color:#b7791f">Benchmark</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">AliceAI-Foundation-80B-A3B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">Qwen3.5-35B-A3B-Base</th>
+<th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #d6a15f;color:#b7791f;font-size:14px">Nemotron-3-Super-120B-A12B-Base</th></tr></thead>
+<tbody>
+<tr><td colspan="4" style="padding:8px 12px;font-weight:600;color:#b7791f;border-bottom:1px solid rgba(239,150,68,0.25);background:rgba(239,150,68,0.1)">Complex reasoning</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">AIME 2026 pass@32</summary><div style="padding-top:6px;line-height:1.4">0-shot problems from the American Invitational Mathematics Examination.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>96.7</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>96.7</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">90.0</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">HMMT 2026 Feb pass@32</summary><div style="padding-top:6px;line-height:1.4">0-shot problems from the February Harvard–MIT Mathematics Tournament.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>96.9</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">87.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">66.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">IMO Answerbench pass@8</summary><div style="padding-top:6px;line-height:1.4">0-shot problems from the International Mathematical Olympiad.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>88.7</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">84.5</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">64.5</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">CodeForces CPP pass@8</summary><div style="padding-top:6px;line-height:1.4">0-shot competitive-programming problems from Codeforces in C++.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">68.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>73.7</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">56.6</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">LiveCodeBench v5-6 pass@1</summary><div style="padding-top:6px;line-height:1.4">0-shot open benchmark of challenging programming problems.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>60.4</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">51.9</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">34.7</td></tr>
+<tr><td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128,128,128,0.15)"><details><summary style="cursor:pointer;white-space:normal;font-weight:600;color:#496fa8">LiveCodeBench v5-6 pass@8</summary><div style="padding-top:6px;line-height:1.4">0-shot open benchmark of challenging programming problems.</div></details></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)"><strong>82.9</strong></td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">82.1</td><td style="padding:7px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15)">59.8</td></tr>
+</tbody>
+</table>
+</div>
+## Usage
+### Transformers
+The model can be run with Transformers. The reference Transformers version is
+5.16.1. Running the KDA layers on GPU requires `flash-linear-attention` with
+KDA support:
+```bash
+pip install \
+  transformers==5.16.1 \
+  accelerate==1.14.0 \
+  flash-linear-attention==0.5.0
+```
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model_id = "yandex/AliceAI-Foundation-80B-A3B-Base"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype=torch.bfloat16,
+    device_map="auto",
+)
+prompt = "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+output_ids = model.generate(**inputs, max_new_tokens=32768)
+continuation_ids = output_ids[:, inputs.input_ids.shape[1] :]
+print(tokenizer.decode(continuation_ids[0], skip_special_tokens=True))
+```
+### vLLM
+The model can also be run with vLLM. Docker and NVIDIA Container Toolkit are
+required.
+```bash
+docker run --name alice-vllm --pull=always --gpus '"device=0,1,2,3"' --ipc=host \
+  -p 8001:8000 \
+  yamlbrand/alice-ai-vllm:latest \
+  yandex/AliceAI-Foundation-80B-A3B-Base \
+  --tensor-parallel-size 4 \
+  --max-model-len auto \
+  --attention-backend FLASH_ATTN \
+  --attention-config.flash_attn_version=2 \
+  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'
+```
+To restart the stopped container while preserving its cache:
+```bash
+docker start -a alice-vllm
+```
+To use all available GPUs, replace `--gpus '"device=0,1,2,3"'` with
+`--gpus all` and set the tensor-parallel size accordingly.
+Once the server is running, send a request:
+```bash
+curl http://127.0.0.1:8001/v1/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "yandex/AliceAI-Foundation-80B-A3B-Base",
+    "prompt": "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?",
+    "max_tokens": 32768,
+    "temperature": 0
+  }'
+```
+### Tokenizer
+The tokenizer is loaded as `LlamaTokenizer` from `tokenizer.model` and uses
+SentencePiece BPE. The `[COT_ENABLE]`, `[COT_START]`, and `[COT_END]` markers,
+as well as the tool-use markers, are ordinary atomic vocabulary tokens rather
+than Hugging Face special tokens.
+In `tokenizer_config.json`, `legacy` is explicitly set to `false` to preserve
+the expected whitespace handling. Do not override it with `true`.
+### Fine-tuning for your tasks
+#### Data format
+To prepare the agentic data used during model training, we used the standard
+OpenAI Messages format. A trajectory is represented as a sequence of messages
+with the `system`, `user`, `assistant`, `tool`, and `meta` roles, while the
+definitions of the available tools are passed separately in the `tools` field.
+Before tokenization, each trajectory was rendered with
+[`chat_template.jinja`](finetune/chat_template.jinja). The template defines the
+role prefixes and the representation of reasoning traces, tool descriptions,
+function calls, and tool results. This is the textual representation in which
+the model encountered these data during training.
+For sft and RL, we recommend storing data in the OpenAI
+Messages format and rendering it with this template. This keeps the new data
+consistent with the format seen by the model during pretraining.
+We intentionally do not set this template as `chat_template` in
+`tokenizer_config.json`: Alice-AI-Foundation-80B-A3B-Base is a base model and therefore does
+not have a single conversational format that should be applied automatically
+during inference. The provided template is intended specifically for preparing
+fine-tuning data.
+#### LoRA fine-tuning example
+The repository includes a minimal PEFT fine-tuning example,
+[`finetune_lora.py`](finetune/finetune_lora.py). It loads a pinned revision of
+the `tatsu-lab/alpaca` dataset, computes the training loss only on responses,
+and saves only the LoRA adapter. A model of this size requires FSDP2; the
+example below is designed for four GPUs with 80 GB of memory each.
+```bash
+pip install \
+  transformers==5.16.1 \
+  accelerate==1.14.0 \
+  peft==0.20.0 \
+  datasets==5.0.1 \
+  flash-linear-attention==0.5.0
+pip install flash-attn==2.8.1 --no-build-isolation
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+accelerate launch \
+  --use_fsdp \
+  --num_processes 4 \
+  --num_machines 1 \
+  --dynamo_backend no \
+  --mixed_precision no \
+  --fsdp_version 2 \
+  --fsdp_reshard_after_forward true \
+  --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP \
+  --fsdp_transformer_layer_cls_to_wrap AliceAIDecoderLayer \
+  --fsdp_cpu_ram_efficient_loading true \
+  --fsdp_sync_module_states true \
+  --fsdp_state_dict_type SHARDED_STATE_DICT \
+  finetune/finetune_lora.py \
+  --model yandex/AliceAI-Foundation-80B-A3B-Base \
+  --steps 100 \
+  --sequence-length 512 \
+  --output-dir alice-lora
+```
+Here, `--mixed_precision no` does not mean that the model uses FP32: the base
+weights are loaded in BF16, while PEFT stores the LoRA parameters in FP32.
+With RAM-efficient loading, only rank 0 loads the full checkpoint weights. The
+other processes construct the model on the meta device and receive their shards
+through FSDP2.

assets/benchmarks.png ADDED Viewed

Git LFS Details

SHA256: ccf544ac6032cdbc1e15b15c8cc1f6dfd086690d356b37f916e9019022974e27
Pointer size: 131 Bytes
Size of remote file: 125 kB

config.json ADDED Viewed

	@@ -0,0 +1,58 @@

+{
+  "architectures": [
+    "AliceAIForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "auto_map": {
+    "AutoConfig": "configuration_alice_ai.AliceAIConfig",
+    "AutoModelForCausalLM": "modeling_alice_ai.AliceAIForCausalLM"
+  },
+  "block_attn_res_block_size": 4,
+  "head_dim": 256,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "kda_allow_negative_eigenvalues": false,
+  "layer_types": [
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention",
+    "linear_attention", "linear_attention", "linear_attention", "full_attention"
+  ],
+  "eos_token_id": 2,
+  "linear_conv_kernel_dim": 4,
+  "linear_key_head_dim": 128,
+  "linear_num_key_heads": 32,
+  "linear_num_value_heads": 32,
+  "linear_value_head_dim": 128,
+  "max_position_embeddings": 262144,
+  "model_type": "alice_ai",
+  "moe_intermediate_size": 512,
+  "mtp_num_hidden_layers": 1,
+  "num_attention_heads": 16,
+  "number_of_conv_states": 3,
+  "num_experts": 512,
+  "num_experts_per_tok": 10,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 2,
+  "output_router_logits": false,
+  "partial_rotary_factor": 0.25,
+  "rms_norm_eps": 1e-6,
+  "rope_theta": 1000000.0,
+  "router_bias_correction": true,
+  "router_score_function": "sigmoid",
+  "shared_expert_intermediate_size": 512,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.16.1",
+  "use_cache": true,
+  "vocab_size": 129024
+}

configuration_alice_ai.py ADDED Viewed

	@@ -0,0 +1,109 @@

+from transformers.configuration_utils import PretrainedConfig
+class AliceAIConfig(PretrainedConfig):
+    model_type = "alice_ai"
+    def __init__(
+        self,
+        vocab_size: int = 129024,
+        hidden_size: int = 2048,
+        num_hidden_layers: int = 48,
+        num_attention_heads: int = 16,
+        num_key_value_heads: int = 2,
+        head_dim: int = 256,
+        linear_num_key_heads: int = 32,
+        linear_num_value_heads: int = 32,
+        linear_key_head_dim: int = 128,
+        linear_value_head_dim: int = 128,
+        linear_conv_kernel_dim: int = 4,
+        num_experts: int = 512,
+        num_experts_per_tok: int = 10,
+        moe_intermediate_size: int = 512,
+        shared_expert_intermediate_size: int = 512,
+        block_attn_res_block_size: int = 4,
+        router_score_function: str = "sigmoid",
+        router_bias_correction: bool = True,
+        kda_allow_negative_eigenvalues: bool = False,
+        max_position_embeddings: int = 262144,
+        rope_theta: float = 1_000_000.0,
+        partial_rotary_factor: float = 0.25,
+        rms_norm_eps: float = 1e-6,
+        hidden_act: str = "silu",
+        initializer_range: float = 0.02,
+        attention_dropout: float = 0.0,
+        use_cache: bool = True,
+        output_router_logits: bool = False,
+        layer_types: list[str] | None = None,
+        tie_word_embeddings: bool = False,
+        pad_token_id: int | None = None,
+        bos_token_id: int | None = None,
+        eos_token_id: int | list[int] | None = None,
+        **kwargs,
+    ) -> None:
+        if layer_types is None:
+            layer_types = [
+                "full_attention" if (layer_idx + 1) % 4 == 0 else "linear_attention"
+                for layer_idx in range(num_hidden_layers)
+            ]
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.head_dim = head_dim
+        self.linear_num_key_heads = linear_num_key_heads
+        self.linear_num_value_heads = linear_num_value_heads
+        self.linear_key_head_dim = linear_key_head_dim
+        self.linear_value_head_dim = linear_value_head_dim
+        self.linear_conv_kernel_dim = linear_conv_kernel_dim
+        self.num_experts = num_experts
+        self.num_experts_per_tok = num_experts_per_tok
+        self.moe_intermediate_size = moe_intermediate_size
+        self.shared_expert_intermediate_size = shared_expert_intermediate_size
+        self.block_attn_res_block_size = block_attn_res_block_size
+        self.router_score_function = router_score_function
+        self.router_bias_correction = router_bias_correction
+        self.kda_allow_negative_eigenvalues = kda_allow_negative_eigenvalues
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.partial_rotary_factor = partial_rotary_factor
+        self.rms_norm_eps = rms_norm_eps
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.attention_dropout = attention_dropout
+        self.use_cache = use_cache
+        self.output_router_logits = output_router_logits
+        self.layer_types = layer_types
+        self.number_of_conv_states = 3
+        self._validate_fields()
+    def _validate_fields(self) -> None:
+        if self.block_attn_res_block_size <= 0:
+            raise ValueError("block_attn_res_block_size must be positive")
+        if self.linear_conv_kernel_dim < 2:
+            raise ValueError("linear_conv_kernel_dim must be at least 2")
+        if self.router_score_function != "sigmoid":
+            raise ValueError("This architecture requires sigmoid routing")
+        if not 0 < self.num_experts_per_tok <= self.num_experts:
+            raise ValueError("num_experts_per_tok must be between 1 and num_experts")
+        if self.num_attention_heads % self.num_key_value_heads != 0:
+            raise ValueError(
+                "num_attention_heads must be divisible by num_key_value_heads"
+            )
+        if self.linear_num_value_heads % self.linear_num_key_heads != 0:
+            raise ValueError(
+                "linear_num_value_heads must be divisible by linear_num_key_heads"
+            )
+        if len(self.layer_types) != self.num_hidden_layers:
+            raise ValueError("layer_types must contain one entry per hidden layer")
+        unknown = set(self.layer_types) - {"linear_attention", "full_attention"}
+        if unknown:
+            raise ValueError(f"Unsupported layer types: {sorted(unknown)}")

finetune/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,94 @@

+{%- set roles = {"assistant": "assistant:", "user": "user:", "system": "system:", "meta": "meta:"} -%}
+{%- set TOOL_CALL_START = "[TOOL_CALL_START]" -%}
+{%- set COT_START = "[COT_START]" -%}
+{%- set COT_END = "[COT_END]" -%}
+{%- set tools_prefix = "Тебе доступны следующие функции:" -%}
+{%- set json_seps = (",", ":") -%}
+{%- macro js(x) -%}
+    {{- x | tojson(separators=json_seps) | replace("&", "\\u0026") | replace("<", "\\u003c") | replace(">", "\\u003e") | replace("'", "\\u0027") | replace("/", "\\/") -}}
+{%- endmacro -%}
+{%- macro tool_definition(tool) -%}
+    {%- if tool.function is defined -%}
+        {%- set tool = tool.function -%}
+    {%- endif -%}
+    {{- 'function {"name":"' ~ tool.name ~ '",' -}}
+    {%- if tool.description is defined and tool.description -%}
+        {{- '"description":"' ~ tool.description ~ '",' -}}
+    {%- endif -%}
+    {{- '"parameters":' ~ js(tool.parameters) -}}
+    {%- if tool.return_parameters is defined and tool.return_parameters -%}
+        {{- ',"return_parameters":' ~ js(tool.return_parameters) -}}
+    {%- endif -%}
+    {%- if tool.few_shot_examples is defined and tool.few_shot_examples -%}
+        {{- ',"few_shot_examples":' ~ js(tool.few_shot_examples) -}}
+    {%- endif -%}
+    {{- "}" -}}
+{%- endmacro -%}
+{%- macro render_tools(tools) -%}
+    {{- tools_prefix -}}
+    {%- for tool in tools -%}
+        {{- "\n" ~ tool_definition(tool) -}}
+    {%- endfor -%}
+{%- endmacro -%}
+{%- macro render_tool_calls(calls) -%}
+    {%- for call in calls -%}
+        {%- set fn = call.function if call.function is defined else call -%}
+        {{- "\n" ~ TOOL_CALL_START ~ fn.name ~ "\n" -}}
+        {%- if fn.arguments is mapping -%}
+            {{- js(fn.arguments) -}}
+        {%- else -%}
+            {{- fn.arguments -}}
+        {%- endif -%}
+    {%- endfor -%}
+{%- endmacro -%}
+{%- macro render_assistant(message) -%}
+    {%- if message.tool_calls is defined and message.tool_calls -%}
+        {%- set content = render_tool_calls(message.tool_calls) -%}
+    {%- elif message.content is defined and message.content is not none -%}
+        {%- set content = message.content -%}
+    {%- else -%}
+        {%- set content = "" -%}
+    {%- endif -%}
+    {{- roles.assistant -}}
+    {%- if message.reasoning_content is defined and message.reasoning_content -%}
+        {{- COT_START ~ message.reasoning_content ~ COT_END -}}
+    {%- endif -%}
+    {{- " " ~ content -}}
+{%- endmacro -%}
+{%- macro render_message(message) -%}
+    {%- if message.role == "system" -%}
+        {{- roles.system ~ " " ~ message.content -}}
+    {%- elif message.role == "user" -%}
+        {{- roles.user ~ " " ~ message.content -}}
+    {%- elif message.role == "assistant" -%}
+        {{- render_assistant(message) -}}
+    {%- elif message.role == "tool" -%}
+        {{- "Tool " ~ message.name ~ ": " ~ (message.content | default("", true)) -}}
+    {%- elif message.role == "meta" -%}
+        {{- roles.meta ~ message.content -}}
+    {%- endif -%}
+{%- endmacro -%}
+{{- bos_token | default("", true) -}}
+{%- if tools is defined and tools -%}
+    {{- render_tools(tools) -}}
+{%- endif -%}
+{%- for message in messages -%}
+    {%- set sep = "\n\n " if (tools is defined and tools) or not loop.first else "" -%}
+    {%- set block = render_message(message) -%}
+    {%- if loop.last and not (add_generation_prompt | default(false)) -%}
+        {%- set block = block | trim -%}
+    {%- endif -%}
+    {%- if block -%}
+        {{- sep ~ block -}}
+    {%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt | default(false) -%}
+    {{- "\n\n " ~ roles.assistant -}}
+{%- endif -%}

finetune/finetune_lora.py ADDED Viewed

	@@ -0,0 +1,274 @@

+from __future__ import annotations
+import argparse
+from pathlib import Path
+import torch
+from accelerate import Accelerator, DistributedType
+from datasets import load_dataset
+from torch.distributed.tensor import DTensor
+from torch.utils.data import DataLoader
+from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
+from peft import LoraConfig, TaskType, get_peft_model, get_peft_model_state_dict
+DEFAULT_MODEL = Path(__file__).resolve().parent.parent
+DATASET_NAME = "tatsu-lab/alpaca"
+DATASET_REVISION = "dce01c9b08f87459cf36a430d809084718273017"
+FSDP_VERSION = 2
+LORA_TARGET_MODULES = ["q_proj", "k_proj", "v_proj", "o_proj"]
+MAX_RESPONSE_TOKENS = 128
+ROUTER_BUFFER_NAME = "e_score_correction_bias"
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="LoRA fine-tuning for Alice AI")
+    parser.add_argument("--model", default=str(DEFAULT_MODEL))
+    parser.add_argument("--output-dir", type=Path)
+    parser.add_argument("--steps", type=int, default=1)
+    parser.add_argument("--sequence-length", type=int, default=32)
+    parser.add_argument("--learning-rate", type=float, default=2e-4)
+    parser.add_argument("--lora-rank", type=int, default=8)
+    parser.add_argument("--seed", type=int, default=42)
+    return parser.parse_args()
+def encode_alpaca_row(row, tokenizer, sequence_length: int) -> dict[str, list[int]]:
+    prompt = f"Instruction:\n{row['instruction']}"
+    if row["input"]:
+        prompt += f"\n\nInput:\n{row['input']}"
+    prompt += "\n\nResponse:\n"
+    prompt_ids = tokenizer(prompt, add_special_tokens=False).input_ids
+    response_ids = tokenizer(row["output"], add_special_tokens=False).input_ids
+    response_budget = min(MAX_RESPONSE_TOKENS, max(1, sequence_length // 4))
+    response_ids = response_ids[:response_budget]
+    prompt_ids = prompt_ids[: sequence_length - len(response_ids) - 2]
+    input_ids = [
+        tokenizer.bos_token_id,
+        *prompt_ids,
+        *response_ids,
+        tokenizer.eos_token_id,
+    ]
+    labels = [-100] * (len(prompt_ids) + 1) + [*response_ids, tokenizer.eos_token_id]
+    attention_mask = [1] * len(input_ids)
+    padding_length = sequence_length - len(input_ids)
+    input_ids.extend([tokenizer.pad_token_id] * padding_length)
+    labels.extend([-100] * padding_length)
+    attention_mask.extend([0] * padding_length)
+    return {
+        "input_ids": input_ids,
+        "attention_mask": attention_mask,
+        "labels": labels,
+    }
+def build_dataloader(tokenizer, sequence_length: int) -> DataLoader:
+    if tokenizer.pad_token_id is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    dataset = load_dataset(
+        DATASET_NAME,
+        revision=DATASET_REVISION,
+        split="train",
+    )
+    dataset = dataset.map(
+        encode_alpaca_row,
+        fn_kwargs={"tokenizer": tokenizer, "sequence_length": sequence_length},
+        remove_columns=dataset.column_names,
+    )
+    return DataLoader(dataset.with_format("torch"), batch_size=1, shuffle=True)
+def save_adapter(accelerator: Accelerator, model, tokenizer, output_dir: Path) -> None:
+    unwrapped_model = accelerator.unwrap_model(model)
+    adapter_state = get_peft_model_state_dict(unwrapped_model)
+    adapter_state = {
+        name: value.full_tensor().cpu() if isinstance(value, DTensor) else value.cpu()
+        for name, value in adapter_state.items()
+    }
+    if accelerator.is_main_process:
+        unwrapped_model.save_pretrained(output_dir, state_dict=adapter_state)
+        tokenizer.save_pretrained(output_dir)
+    accelerator.wait_for_everyone()
+def restore_rotary_buffer(
+    accelerator: Accelerator,
+    model,
+    device: torch.device | None = None,
+) -> None:
+    unwrapped_model = accelerator.unwrap_model(model)
+    config = unwrapped_model.config
+    base_model = (
+        unwrapped_model.get_base_model()
+        if hasattr(unwrapped_model, "get_base_model")
+        else unwrapped_model
+    )
+    rotary_embedding = base_model.model.rotary_emb
+    rotary_dim = int(config.head_dim * config.partial_rotary_factor)
+    inv_freq = 1.0 / (
+        config.rope_theta
+        ** (
+            torch.arange(
+                0,
+                rotary_dim,
+                2,
+                dtype=torch.float32,
+                device=device or rotary_embedding.inv_freq.device,
+            )
+            / rotary_dim
+        )
+    )
+    rotary_embedding.inv_freq = inv_freq
+def load_model(model_path: str, accelerator: Accelerator):
+    load_kwargs = {
+        "trust_remote_code": True,
+        "dtype": torch.bfloat16,
+        "attn_implementation": "flash_attention_2",
+    }
+    if (
+        not accelerator.state.fsdp_plugin.cpu_ram_efficient_loading
+        or accelerator.is_main_process
+    ):
+        return AutoModelForCausalLM.from_pretrained(model_path, **load_kwargs)
+    config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
+    # Transformers validates FlashAttention against a real device even though
+    # the attention backend does not affect the module layout.
+    config._attn_implementation = "eager"
+    previous_dtype = torch.get_default_dtype()
+    torch.set_default_dtype(load_kwargs["dtype"])
+    try:
+        with torch.device("meta"):
+            model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)
+    finally:
+        torch.set_default_dtype(previous_dtype)
+    model.config._attn_implementation = load_kwargs["attn_implementation"]
+    return model
+def remove_router_buffers(model, is_main_process: bool) -> dict[str, torch.Tensor]:
+    router_buffers = {}
+    for module_name, module in model.named_modules():
+        buffer = module._buffers.pop(ROUTER_BUFFER_NAME, None)
+        if buffer is None:
+            continue
+        router_buffers[module_name] = (
+            buffer.detach().cpu()
+            if is_main_process
+            else torch.empty(buffer.shape, dtype=buffer.dtype)
+        )
+    return router_buffers
+def restore_router_buffers(
+    accelerator: Accelerator,
+    model,
+    router_buffers: dict[str, torch.Tensor],
+) -> None:
+    unwrapped_model = accelerator.unwrap_model(model)
+    for module_name, buffer in router_buffers.items():
+        device_buffer = buffer.to(accelerator.device)
+        torch.distributed.broadcast(device_buffer, src=0)
+        unwrapped_model.get_submodule(module_name).register_buffer(
+            ROUTER_BUFFER_NAME,
+            device_buffer,
+        )
+def materialize_trainable_parameters(model) -> None:
+    # Accelerate 1.14 remaps FSDP2 optimizer parameters by data_ptr(). All meta
+    # tensors use pointer 0, so give the small trainable LoRA tensors real storage.
+    for module in model.modules():
+        for parameter_name, parameter in tuple(module.named_parameters(recurse=False)):
+            if parameter.requires_grad and parameter.is_meta:
+                module._parameters[parameter_name] = torch.nn.Parameter(
+                    torch.empty_like(parameter, device="cpu"),
+                )
+def main() -> None:
+    args = parse_args()
+    if args.steps < 1:
+        raise ValueError("--steps must be positive")
+    if args.sequence_length <= 1:
+        raise ValueError("--sequence-length must be at least 2")
+    accelerator = Accelerator()
+    if accelerator.distributed_type != DistributedType.FSDP:
+        raise RuntimeError("Launch this script with Accelerate FSDP")
+    if accelerator.state.fsdp_plugin.fsdp_version != FSDP_VERSION:
+        raise RuntimeError("This example requires FSDP2")
+    torch.manual_seed(args.seed)
+    tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
+    with accelerator.main_process_first():
+        dataloader = build_dataloader(tokenizer, args.sequence_length)
+    lora_config = LoraConfig(
+        task_type=TaskType.CAUSAL_LM,
+        r=args.lora_rank,
+        lora_alpha=2 * args.lora_rank,
+        lora_dropout=0.0,
+        target_modules=LORA_TARGET_MODULES,
+        bias="none",
+    )
+    model = load_model(args.model, accelerator)
+    if accelerator.state.fsdp_plugin.cpu_ram_efficient_loading:
+        restore_rotary_buffer(accelerator, model, device=torch.device("cpu"))
+    model.config.use_cache = False
+    model = get_peft_model(model, lora_config)
+    if accelerator.state.fsdp_plugin.cpu_ram_efficient_loading:
+        materialize_trainable_parameters(model)
+    router_buffers = (
+        remove_router_buffers(model, accelerator.is_main_process)
+        if accelerator.state.fsdp_plugin.cpu_ram_efficient_loading
+        else {}
+    )
+    model.gradient_checkpointing_enable(
+        gradient_checkpointing_kwargs={"use_reentrant": False}
+    )
+    trainable = sum(
+        parameter.numel() for parameter in model.parameters() if parameter.requires_grad
+    )
+    total = sum(parameter.numel() for parameter in model.parameters())
+    accelerator.print(f"Trainable parameters: {trainable:,} / {total:,}")
+    optimizer = torch.optim.AdamW(
+        (parameter for parameter in model.parameters() if parameter.requires_grad),
+        lr=args.learning_rate,
+    )
+    model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
+    restore_router_buffers(accelerator, model, router_buffers)
+    restore_rotary_buffer(accelerator, model)
+    model.train()
+    data_iterator = iter(dataloader)
+    for step in range(args.steps):
+        try:
+            batch = next(data_iterator)
+        except StopIteration:
+            data_iterator = iter(dataloader)
+            batch = next(data_iterator)
+        optimizer.zero_grad(set_to_none=True)
+        outputs = model(**batch, use_cache=False)
+        accelerator.backward(outputs.loss)
+        optimizer.step()
+        accelerator.print(
+            f"step={step + 1} loss={outputs.loss.detach().float().item():.6f}"
+        )
+    if args.output_dir is not None:
+        save_adapter(accelerator, model, tokenizer, args.output_dir)
+    accelerator.print("LoRA fine-tuning smoke test completed")
+    accelerator.end_training()
+if __name__ == "__main__":
+    main()

model-00001-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dc8752372a7af35f5c9385b32a24b15783307f1e56e11902bbca3944e57d68a0
+size 3907510776

model-00002-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2cd4cf968c6ec6d8d8b6c03d6dcc6e93b181bf93b8934b7f9b9cf69fc1310ad1
+size 3300139664

model-00003-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fa2bdd505ea6c76a1ba921bc457133f21e7e9520277dd61a9c3415c08d048205
+size 3284173088

model-00004-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ddd0ff34293459b64fdee66d7fea280ef373c1f0af45560194b254a64dc923e6
+size 3300139664

model-00005-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:78e8cccae89bf63a7123f5a44c575edbcdbc47c049906b17887a01a3b5b85703
+size 3300139664

model-00006-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ab3cdf27eb28f20be5a1854b6ccd297e4c2089f819ec380a2386c2715cbf35da
+size 3300139664

model-00007-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5ee49728c98e1008fb8302573a55cb42c1af8acd104a9deb6f817f1a223f6cf1
+size 3284173088

model-00008-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f9d039f8b7ab5870d74965ecfa6555f730dd03b6feda8cb4dc07f486a24e8638
+size 3300139664

model-00009-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c8acb2ccc31d519bf5814e54d4731ea077f9b4993e86f111dddeabf2d2c633f3
+size 3300139664

model-00010-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2ba4a9bab910a68d0ea6e43035bd7c9041baaf5003671df066cad950ae905738
+size 3300139576

model-00011-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:53b2e775863ade93ee3da92005b09f53bc1b06d6348df76424178ce3ab42cf30
+size 3284173104

model-00012-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fc8d2867f1ca4c018caa0d0520e54b8b73ca1071ad2c883b82ad7c5be307731f
+size 3300139696

model-00013-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6717e4cc976388adfb12dfc2e367c370b2c2d10bb779e7b9527ce8e8a4aaf161
+size 3300139696

model-00014-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5ab428fefdf329131479a8b43fb51878c74cd73f4732c2cd774773f506663afe
+size 3300139696

model-00015-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7946bda09928d412a9dfecac35f85613306c43f677e3004eea5c322ec171bf97
+size 3284173104

model-00016-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4844a9f4d8d5ff7ae063d78e1f6b86ed219b0fe9542c3b6473fe853ad5ac1329
+size 3300139696

model-00017-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:acc60ddc1e922706d75b7583d003f235c2ff43cf656f31d3de0bfa86c5892cee
+size 3300139696

model-00018-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d45cbc29dd70710361ca57a7f89ef6087e04f6fed1b25646bb7d7874ceebdf25
+size 3300139696

model-00019-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ba1eda354b8b364b38b18460254d94304602693e4ad8cc634c48bff187a2ba39
+size 3284173104

model-00020-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7f133b92c9de031f997bfe04f8cf01ed49694ebf32dbb76cc633cf8186c9d172
+size 3300139696

model-00021-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fae72d324834b897fe9973d5fdf94c9a81f2a594fb39b10c41c912e444ec2a69
+size 3300139696

model-00022-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9ba0f129fe6694303eb26385a70d628b0ee88be625ee34d04acecb130c8716e3
+size 3300139696

model-00023-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6d9632bd7065a4f8c4ba884e28492e90aabbce5b9f1c3638ec5d80fd2ce71a35
+size 3284173104

model-00024-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6c937a38a72f9b1e87af22c5a80115b832cf1630485019a56c38bb60d42bcd85
+size 3300139696

model-00025-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8f61093583ce87a5b236a3beb5384ee41a363db3dcbbe3811aa4786bdf5cb072
+size 3300139696

model-00026-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4225886e8dbf1e07a75228a2e455d00932c6f537de6f5541d12e7377df0d7872
+size 3300139696

model-00027-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ad7a806a91bcc936c35e9190bac39de33871ac11c65471cdc3082fd52e609819
+size 3284173104

model-00028-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ad364d878d318cc67b3e5d5a06405ee5f2065cf66ba908d823483295c35a62b7
+size 3300139696

model-00029-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:24336b293f97861b7442bcdb4c2e9fa68b02554ae7afddccd5c48f9ec79c0a11
+size 3300139696

model-00030-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d7e63e8ffba289606d19eee08c6ddbcc85fb0ed1c8b28f2aa90b3d278bfe49c7
+size 3300139696

model-00031-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:34849f8d875cbc6f360c3ca2e657762963bd15fbefec8dea11b817bb646215bc
+size 3284173104

model-00032-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2388667584c9c5bc9c5ea52453ede54818b87e8036911f3ad68154b14a1fb7e6
+size 3300139696

model-00033-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:15b8af443b13f3e3b9f7509885b262af0a12031b1e1372ac4516f759d4a9e919
+size 3300139696

model-00034-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:785f5e1c7395bd155f8bfaffa114a3a3da6ad3480bc2530a8d901c2016be202a
+size 3300139696

model-00035-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f440c22516776bda8622abb8e62818413b8cbc7c30e43032c6cd858f04282bea
+size 3284173104

model-00036-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2c0b96f186169421ab809742b34cc2d6b36d90ab8a724a68293e5408be3f14a7
+size 3300139696

model-00037-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3554c046833ba5156a2ffe2806789929fe5926f2d3ada3edb4ad605239f72252
+size 3300139696

model-00038-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:27d841a333d25b7f81e342b7d73f058f36fa88aafec9dd61cb5b4e4440942740
+size 3300139696

model-00039-of-00049.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d6a4d680c385d9366642b15346a2bf5fe41fc5e02b097b5d46953843e1b75be4
+size 3284173104