Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

INFERENCE_OPTIMIZATION.md

@@ -0,0 +1,85 @@
+# 完整模型精简与推理加速
+
+精简模型位于 `runs/bilingual_s5_full/final_compact/`，由 `runs/bilingual_s5_full/final/` 导出，保留文本、图像、视频、音频理解和语音输出。原始目录保留。
+
+## 模型体积
+
+| 项目 | 原模型 | 精简模型 |
+|---|---:|---:|
+| 独立参数量 | 2,031,972,995 | 2,031,710,851 |
+| safetensors 权重字节数 | 5,677,248,858 | 4,115,251,826 |
+| 权重文件大小（十进制 GB） | 5.677 | 4.115 |
+
+权重文件减少 27.5%，实际参数减少 262,144（约 0.013%）。参数统计覆盖完整多模态模型，共享参数只计算一次。
+
+删除的是 codec 两个从未参与解码的 `input_proj`。Thinker 按配置的 BF16 推理精度保存；音频投影器和特殊 token 参数保持 FP32。精简目录支持常规 `save_pretrained` / `from_pretrained`。
+
+Talker 的 132,588 行文本 embedding 全部可达且无完全重复行；32 张声码 embedding／输出头也无可进一步共享的完整表或重复行。原有 15 张共享声码 embedding 保持共享。
+
+## 推理结构
+
+- 音频分块长度在 CPU 上一次计算，SDPA 各层复用，避免反复从 GPU 取回长度。
+- 不做音频池化或堆叠时，直接返回编码输出。
+- Thinker 的线性注意力 decoder 层在 CUDA 上对单 token 解码使用 CUDA Graph，覆盖归一化、递归状态更新、MLP 和残差计算。运算、权重和动态 KV cache 保持原样。
+- 每个请求保有独立缓存；训练、有梯度的 forward、CPU 和多 token prefill 执行常规模型路径。改变设备、精度或训练模式会释放已有 graph。
+- `generate()` 使用公开的 `model.generation_config`，并在调用期间选择 Flash／memory-efficient／math SDPA 后端。
+
+## 验证结果
+
+完整 MMAU test-mini 1,000 条输入中，原模型与精简模型生成 token 和文本全部一致，均自然到 EOS；答案字母准确率均为 **64.9%（649/1000）**。sound、speech、music 分别为 240/333、210/333、199/334。清单为 `data/spoken_execution_eval/mmau_test_mini_1000.jsonl`，提示词保持该清单原样。
+
+音频问答时延在 sound／speech／music 各 4 条代表样本上各测 3 次，两模型交替执行，所有 token 一致。MMAU 每条输出 2 token（答案字母和 EOS）。
+
+| 中位数指标 | 原模型 | 精简模型 |
+|---|---:|---:|
+| 整答时延 | 173.03 ms | 97.35 ms |
+| TTFT | 129.12 ms | 86.43 ms |
+| 解码吞吐量 | 30.71 token/s | 65.03 token/s |
+| 端到端吞吐量 | 11.56 token/s | 20.54 token/s |
+
+同一样本成对加速比中位数 **1.82×**。TTFT／解码吞吐量来自独立 streaming 测量，普通整答时延未插入逐 token 同步。
+
+6 个中英文文本任务、每任务 2 次普通测量和 1 次独立 streaming 测量，生成 token 全部一致；回答长度为 10、35、58、110、256、256 token。整答成对加速比中位数 **2.44×**，整答时延中位数 **2.490 s → 0.944 s**，解码吞吐量中位数 **32.41 → 77.00 token/s**。两条 256-token 输出达到设定上限，两模型停止位置相同。
+
+加载后 1,574 个保留 state-dict tensor 逐元素一致；真实音频投影特征以及中英文语音的声码、波形逐元素一致。CPU／CUDA 测试覆盖 FP32/BF16、缓存独立性、保留 hidden states、图复用、训练梯度、保存加载及图像／视频路径，共 **126 项通过**。
+
+结果文件位于 `runs/hf_inference_optimization/`：
+
+- `mmau_equivalence.json`：1,000 条 MMAU 输出及准确率对比。
+- `mmau_verified_shard_0.json` 至 `mmau_verified_shard_3.json`：完整逐条记录。
+- `mmau_latency.json`：12 条样本、3 次重复的最终音频问答计时。
+- `text_decode_pair.json`：长回答计时、TTFT、吞吐量和 token 对比。
+- `weight_and_speech_equivalence.json`：权重、音频特征和语音输出等价验证。
+- `redundancy.json`：embedding 可达性及逐行去重统计。
+
+实测使用 NVIDIA H20、PyTorch 2.11.0+cu130、Transformers 5.12.1、fla-core 0.5.0、causal-conv1d 1.6.0；CPU 功能及独立 CUDA Graph 测试另外覆盖 PyTorch 2.6.0+cu124。
+
+## 使用
+
+上传的 `run_mmau.py` 只需把 `--model` 指向 `runs/bilingual_s5_full/final_compact`。默认自动使用优化路径，保留两次预热。
+
+重新导出：
+
+```bash
+python -m hf.compact_edgeinstant \
+  --source runs/bilingual_s5_full/final \
+  --output runs/bilingual_s5_full/final_compact_new \
+  --device cuda:0
+```
+
+配对测量：
+
+```bash
+OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 python scripts/benchmark_hf_inference.py \
+  --source runs/bilingual_s5_full/final \
+  --candidate runs/bilingual_s5_full/final_compact \
+  --manifest runs/hf_inference_optimization/mmau_pair_manifest.jsonl \
+  --repeats 3 --device cuda:0 \
+  --output runs/hf_inference_optimization/mmau_latency.json
+```
+
+测量包含前处理、设备传输、贪心生成至 EOS／token 上限、文本解码和 CUDA 同步。音频文件读取及重采样在计时外完成。TTFT 在独立的 streaming 测��中从前处理开始，至第一个新 token 返回。解码吞吐量为 `(生成 token 数 − 1) / (最后 token 时间 − 首 token 时间)`，端到端吞吐量为 `生成 token 数 / 整答时延`；均包含 EOS。普通整答计时不插入 streamer。
+
+## 限制
+
+等价基准是原 checkpoint 按上传脚本的 `dtype="auto"` 加载后的模型；精简权重不保留原文件额外的 FP32 小数精度，原始目录可继续用于 FP32 实验。首次 CUDA Graph 建立和新 batch shape 需要预热。最终测量时机器有其他 GPU 工作负载，因此绝对时延受资源争用影响；同一样本的两模型在同一 GPU 上交替执行。绝对速度还取决于硬件、CUDA／PyTorch 和 kernel 依赖，实际运行环境及其他 GPU 进程均记录在测量 JSON 中。MMAU 短答案通常只有答案字母和 EOS，长回答吞吐量应参考独立的文本测试。

LICENSE.codec

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Alibaba Cloud
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -0,0 +1,72 @@
+---
+library_name: transformers
+pipeline_tag: audio-text-to-text
+language:
+- zh
+- en
+tags:
+- edgeinstant
+- audio
+- speech-recognition
+- speech-translation
+- audio-question-answering
+---
+
+# EdgeInstant-1.5b S5 bilingual audio instruction
+
+中英文 audio-in 后训练模型，支持语音转写、语音翻译、音频问答和指令执行，同时保留 direct 与 thinking 两种输出模式。此目录是包含模型权重、处理器及自定义代码的独立 Hugging Face 包。
+
+## 训练
+
+起点为 S4 最终模型，先以相同数据进行 600 步回答控制训练，再进行 1,600 步八卡混合后训练。语言模型全部参数、音频投影层和音频编码器末两层参与主训练；峰值学习率分别为 1e-6、2e-7、1e-7。
+
+主训练清单包含 531,174 行，涵盖中英文 ASR、双向语音翻译、多来源文本指令、真实语音命令提取、语音语境问答、声音和音乐理解、直接回答及思考数据。正监督中 thinking 占 3.76%。通过 EOS 权重 4 和重复负样本 unlikelihood 权重 0.2 改善回答结束及循环输出；推理使用 greedy。
+
+交付权重为主训练第 1,600 步。模型依据独立开发集上的生成稳定性、准确率、ASR 和翻译表现选择。完整配置见仓库 `configs/train/bilingual_s5.yaml`，方法与数据说明见 `docs/BILINGUAL_S5.md`。测试样本仅用于评测。
+
+## 使用
+
+安装本目录 `requirements.txt`。音频输入为 16 kHz 单声道；保留完整波形，不设 30 秒截断。
+
+```python
+import soundfile as sf
+import torch
+from transformers import AutoModel, AutoProcessor
+
+path = "/default-filesys/workspace/chenjunzhe/EdgeInstant-1.5b/runs/bilingual_s5_full/final"
+processor = AutoProcessor.from_pretrained(path, trust_remote_code=True)
+model = AutoModel.from_pretrained(
+    path, trust_remote_code=True, dtype=torch.bfloat16,
+).to("cuda").eval()
+torch.backends.cuda.enable_cudnn_sdp(False)
+
+waveform, sample_rate = sf.read("speech.wav", dtype="float32")
+inputs = processor(
+    audio=waveform, sampling_rate=sample_rate, task="asr",
+    text="请将语音转写为文字，只输出转写结果。", enable_thinking=False,
+).to("cuda")
+with torch.inference_mode():
+    output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+answer = output[0, inputs["input_ids"].shape[1]:]
+print(processor.decode(answer, skip_special_tokens=True))
+```
+
+语音翻译或问答使用 `task="qa"`，并通过 `text` 指定请求；直接执行语音中的指令时，可用“请听取并完成语音中的请求。”。文本输入使用 `processor(text=...)`。需要思考模式时设置 `enable_thinking=True`，为思考和最终答案留足输出 token。要求只输出选项时使用 `enable_thinking=False` 并在请求中明确输出格式。
+
+## 评测
+
+评测结论及已知限制见 [RESULTS.md](../RESULTS.md)，27 项同协议完整测试与 S4 对照见 [final_results.md](../final_results.md)，固定开发集见 [dev_comparison.md](../dev_comparison.md)。CER/WER 和正确率在结果表中使用百分数，BLEU 保持原单位。W&B 评测仅记录数值指标，预测保留在本地。
+
+真人语音命令在官方未见说话人验证集上，每种输出视图各 3,118 条：JSON 对象完全正确率 93.49%，单字段准确率 96.79%。这些结果衡量 FSC 字段约定下的语音命令提取。公开语境问答采用数据集原有参考，部分来源由模型生成标注。
+
+## 能力限制
+
+开放式复杂语音推理、口述数字计算和中译英语音翻译仍有不足。输出终止、格式正确不等于内容正确；thinking 模式也不保证准确率更高。极低音量或信息不足的语音可能产生无依据转写。新增中文口述数字开发集仅一条，不能用于估计整体中文计算能力。
+
+本轮训练与评测针对 audio-in 和文本回答；Talker 与语音合成组件沿用起点权重。模型包含 Qwen 组件，各组件及训练数据的原许可仍适用；解码器代码许可见 `LICENSE.codec`。
+
+## 精简推理版本
+
+本目录保留全部多模态能力，采用 BF16 推理权重、FP32 音频投影器与特殊 token 参数，移除 codec 未使用的输入投影。CUDA 单 token 解码自动对线性注意力 decoder 层使用 CUDA Graph，模型加载与 `generate()` 调用方式不变。原始 checkpoint 按 `dtype="auto"` 加载时，MMAU 1,000 条生成 token 完全一致，准确率均为 64.9%。
+
+参数量、权重体积、TTFT、吞吐量、验证范围和复现命令见 [推理优化说明](INFERENCE_OPTIMIZATION.md)。

chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is true %}
+        {{- '<think>\n' }}
+    {%- else %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

codec_edgeinstant.py

@@ -0,0 +1,233 @@
+# Copyright 2026 The Alibaba Qwen team.
+# SPDX-License-Identifier: Apache-2.0
+"""Qwen3-TTS 12.5 Hz waveform decoder for the EdgeInstant model."""
+
+from __future__ import annotations
+
+import json
+import math
+from pathlib import Path
+
+import torch
+from torch import nn
+from torch.nn import functional as F
+from transformers.models.qwen3_omni_moe.configuration_qwen3_omni_moe import (
+    Qwen3OmniMoeCode2WavConfig,
+)
+from transformers.models.qwen3_omni_moe.modeling_qwen3_omni_moe import (
+    Qwen3OmniMoeCausalConvNet,
+    Qwen3OmniMoeCode2WavDecoderResidualUnit,
+    Qwen3OmniMoeCode2WavTransformerModel,
+    Qwen3OmniMoeConvNeXtBlock,
+    SnakeBeta,
+)
+
+
+class _CausalTransConvNet(nn.Module):
+    def __init__(self, in_channels, out_channels, kernel_size, stride=1):
+        super().__init__()
+        self.conv = nn.ConvTranspose1d(in_channels, out_channels, kernel_size, stride=stride)
+        self.right_pad = kernel_size - stride
+
+    def forward(self, hidden):
+        hidden = self.conv(hidden)
+        if self.right_pad:
+            hidden = hidden[..., : -self.right_pad]
+        return hidden.contiguous()
+
+
+class _Codebook(nn.Module):
+    def __init__(self, dim, size):
+        super().__init__()
+        self.cluster_usage = nn.Parameter(torch.ones(size))
+        self.embedding_sum = nn.Parameter(torch.zeros(size, dim))
+
+    def forward(self, codes):
+        embedding = self.embedding_sum / self.cluster_usage.clamp(min=1e-5)[:, None]
+        return F.embedding(codes, embedding)
+
+
+class _VectorQuantization(nn.Module):
+    def __init__(self, dim, size):
+        super().__init__()
+        self._codebook = _Codebook(dim, size)
+
+    def forward(self, codes):
+        return self._codebook(codes).transpose(1, 2)
+
+
+class _ResidualVectorQuantization(nn.Module):
+    def __init__(self, count, dim, size):
+        super().__init__()
+        self.layers = nn.ModuleList([_VectorQuantization(dim, size) for _ in range(count)])
+
+    def forward(self, codes):
+        quantized = torch.zeros([1], device=codes.device)[0]
+        for index, layer in enumerate(self.layers):
+            quantized = quantized + layer(codes[:, index])
+        return quantized
+
+
+class _ResidualVectorQuantizer(nn.Module):
+    def __init__(self, count, config, *, inference_only=False):
+        super().__init__()
+        dim = config.codebook_dim // 2
+        self.input_proj = (
+            None if inference_only else nn.Conv1d(config.codebook_dim, dim, 1, bias=False)
+        )
+        self.output_proj = nn.Conv1d(dim, config.codebook_dim, 1, bias=False)
+        self.vq = _ResidualVectorQuantization(count, dim, config.codebook_size)
+
+    def forward(self, codes):
+        return self.output_proj(self.vq(codes))
+
+
+class _SplitResidualVectorQuantizer(nn.Module):
+    def __init__(self, config, *, inference_only=False):
+        super().__init__()
+        self.rvq_first = _ResidualVectorQuantizer(1, config, inference_only=inference_only)
+        self.rvq_rest = _ResidualVectorQuantizer(
+            config.num_quantizers - 1, config, inference_only=inference_only,
+        )
+
+    def forward(self, codes):
+        quantized = self.rvq_first(codes[:, :1])
+        quantized += self.rvq_rest(codes[:, 1:])
+        return quantized
+
+
+class _ProjectedTransformer(Qwen3OmniMoeCode2WavTransformerModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.input_proj = nn.Linear(config.latent_dim, config.hidden_size)
+        self.output_proj = nn.Linear(config.hidden_size, config.latent_dim)
+
+    def forward(self, hidden):
+        positions = torch.arange(hidden.shape[1], device=hidden.device)
+        distances = positions[:, None] - positions[None, :]
+        mask = ((distances >= 0) & (distances < self.config.sliding_window))[None, None]
+        output = super().forward(
+            inputs_embeds=self.input_proj(hidden),
+            attention_mask={"sliding_attention": mask.expand(hidden.shape[0], -1, -1, -1)},
+            use_cache=False,
+        )
+        return self.output_proj(output.last_hidden_state)
+
+
+class _DecoderBlock(nn.Module):
+    def __init__(self, config, index):
+        super().__init__()
+        input_dim = config.decoder_dim // 2**index
+        output_dim = input_dim // 2
+        rate = config.upsample_rates[index]
+        self.block = nn.ModuleList(
+            [SnakeBeta(input_dim), _CausalTransConvNet(input_dim, output_dim, 2 * rate, rate)]
+            + [Qwen3OmniMoeCode2WavDecoderResidualUnit(output_dim, dilation) for dilation in (1, 3, 9)]
+        )
+
+    def forward(self, hidden):
+        for block in self.block:
+            hidden = block(hidden)
+        return hidden
+
+
+class EdgeInstantCodecDecoder(nn.Module):
+    """Decode integer codes ``[batch, frames, 16]`` to float audio ``[batch, samples]``.
+
+    ``config`` is the complete Qwen3-TTS speech tokenizer configuration. Weights
+    match its ``decoder.*`` tensors after removing that prefix. Setting its
+    top-level ``inference_only`` field omits the unused quantizer input projections.
+    """
+
+    def __init__(self, config: dict):
+        super().__init__()
+        self.sample_rate = int(config["output_sample_rate"])
+        self.samples_per_frame = int(config["decode_upsample_rate"])
+        decoder_config = dict(config["decoder_config"])
+        decoder_config["rope_parameters"] = {
+            "rope_type": "default",
+            "rope_theta": decoder_config.pop("rope_theta", 10000),
+        }
+        self.config = Qwen3OmniMoeCode2WavConfig(**decoder_config)
+        self.config._attn_implementation = "sdpa"
+        cfg = self.config
+        if math.prod([*cfg.upsample_rates, *cfg.upsampling_ratios]) != self.samples_per_frame:
+            raise ValueError("decode_upsample_rate must match the waveform decoder upsampling factors")
+        self.pre_transformer = _ProjectedTransformer(cfg)
+        self.quantizer = _SplitResidualVectorQuantizer(
+            cfg, inference_only=bool(config.get("inference_only", False)),
+        )
+        self.pre_conv = Qwen3OmniMoeCausalConvNet(cfg.codebook_dim, cfg.latent_dim, 3)
+        self.upsample = nn.ModuleList(
+            [
+                nn.ModuleList(
+                    [
+                        _CausalTransConvNet(cfg.latent_dim, cfg.latent_dim, factor, factor),
+                        Qwen3OmniMoeConvNeXtBlock(cfg.latent_dim),
+                    ]
+                )
+                for factor in cfg.upsampling_ratios
+            ]
+        )
+        output_dim = cfg.decoder_dim // 2 ** len(cfg.upsample_rates)
+        self.decoder = nn.ModuleList(
+            [Qwen3OmniMoeCausalConvNet(cfg.latent_dim, cfg.decoder_dim, 7)]
+            + [_DecoderBlock(cfg, index) for index in range(len(cfg.upsample_rates))]
+            + [SnakeBeta(output_dim), Qwen3OmniMoeCausalConvNet(output_dim, 1, 7)]
+        )
+
+    def forward(self, codes: torch.Tensor) -> torch.Tensor:
+        if codes.ndim != 3 or codes.shape[-1] != self.config.num_quantizers:
+            raise ValueError(f"codes must be [batch, frames, {self.config.num_quantizers}]")
+        if codes.shape[1] == 0:
+            return self.pre_conv.conv.weight.new_empty((codes.shape[0], 0), dtype=torch.float32)
+        codes = codes.to(device=self.pre_conv.conv.weight.device, dtype=torch.long)
+        hidden = self.quantizer(codes.transpose(1, 2))
+        hidden = self.pre_conv(hidden).transpose(1, 2)
+        hidden = self.pre_transformer(hidden).transpose(1, 2)
+        for blocks in self.upsample:
+            for block in blocks:
+                hidden = block(hidden)
+        for block in self.decoder:
+            hidden = block(hidden)
+        return hidden.squeeze(1).clamp(min=-1, max=1).float()
+
+    def decode(
+        self, codes: torch.Tensor, *, chunk_size: int = 300, left_context_frames: int = 25
+    ) -> torch.Tensor:
+        """Decode with the speech tokenizer's chunking and left context convention."""
+        if chunk_size <= 0 or left_context_frames < 0:
+            raise ValueError("chunk_size must be positive and left_context_frames nonnegative")
+        if codes.shape[1] == 0:
+            return self(codes)
+        waves = []
+        for start in range(0, codes.shape[1], chunk_size):
+            context = min(start, left_context_frames)
+            wave = self(codes[:, start - context : start + chunk_size])
+            waves.append(wave[:, context * self.samples_per_frame :])
+        return torch.cat(waves, dim=-1)
+
+
+def load_codec_decoder(
+    directory: str | Path, *, dtype: torch.dtype = torch.float32, device: str = "cpu"
+) -> EdgeInstantCodecDecoder:
+    """Load the decoder tensors from a local Qwen3-TTS speech tokenizer directory."""
+    from safetensors import safe_open
+
+    directory = Path(directory)
+    with (directory / "config.json").open() as handle:
+        config = json.load(handle)
+    decoder = EdgeInstantCodecDecoder(config).to(dtype=dtype, device=device)
+    rotary = decoder.pre_transformer.rotary_emb
+    rotary.inv_freq, rotary.attention_scaling = rotary.compute_default_rope_parameters(
+        rotary.config, torch.device(device)
+    )
+    rotary.original_inv_freq = rotary.inv_freq.clone()
+    with safe_open(directory / "model.safetensors", framework="pt", device="cpu") as checkpoint:
+        state = {
+            key.removeprefix("decoder."): checkpoint.get_tensor(key)
+            for key in checkpoint.keys()
+            if key.startswith("decoder.")
+        }
+    decoder.load_state_dict(state, strict=True)
+    return decoder.eval()

compact_native_clock.py

@@ -0,0 +1,174 @@
+"""Compact inference representation of the frozen native-token acoustic model."""
+from __future__ import annotations
+
+import copy
+from pathlib import Path
+
+import torch
+from torch import nn
+
+from .native_clock_talker import NativeClockTalker
+from .residual_predictor import Qwen3TTSResidualPredictor
+
+
+class ProjectedTextEmbedding(nn.Module):
+    """Keep projected rows reachable from the complete native byte vocabulary."""
+
+    def __init__(self, weight: torch.Tensor, teacher_ids: torch.Tensor, vocabulary_size: int):
+        super().__init__()
+        self.embedding = nn.Embedding.from_pretrained(weight, freeze=True)
+        rows = torch.full((vocabulary_size,), -1, device=weight.device, dtype=torch.long)
+        rows[teacher_ids.to(weight.device)] = torch.arange(len(teacher_ids), device=weight.device)
+        self.register_buffer("teacher_to_row", rows)
+
+    def forward(self, teacher_ids: torch.Tensor) -> torch.Tensor:
+        return self.embedding(self.teacher_to_row[teacher_ids])
+
+
+class CompactNativeClockTalker(NativeClockTalker):
+    """Fold frozen text projection and share identical codec input embeddings."""
+
+    serialization_buffers = ("prefix", "prefix_english", "prefix_auto", "tts_eos", "tts_pad")
+    tied_weight_keys = {
+        "residual_predictor.q0_embedding.weight": "codec_history_embeddings.0.weight",
+        **{
+            f"residual_predictor.code_embeddings.{group}.weight":
+                f"codec_history_embeddings.{group + 1}.weight"
+            for group in range(14)
+        },
+    }
+
+    def prepare_for_serialization(self) -> "CompactNativeClockTalker":
+        """Save speaker conditions and the runtime precision of rotary frequencies."""
+        self._non_persistent_buffers_set.difference_update(self.serialization_buffers)
+        for backbone in (self.backbone, self.residual_predictor.backbone):
+            backbone.rotary_emb._non_persistent_buffers_set.difference_update(
+                ("inv_freq", "original_inv_freq")
+            )
+        return self
+
+    def get_config(self) -> dict:
+        """Describe the compact model without paths to its initialization assets."""
+        residual_config = self.residual_predictor.backbone.config.to_dict()
+        residual_config["rope_theta"] = self.residual_predictor.backbone.config.rope_parameters["rope_theta"]
+        return {
+            "backbone_config": self.backbone.config.to_dict(),
+            "residual_predictor_config": residual_config,
+            "attn_implementation": self.backbone.config._attn_implementation,
+            "hidden_size": self.hidden_size,
+            "codebook_size": self.codebook_size,
+            "num_code_groups": len(self.codec_history_embeddings),
+            "projected_num_embeddings": self.text_embedding.embedding.num_embeddings,
+            "teacher_vocab_size": self.text_embedding.teacher_to_row.numel(),
+            "prefix_shapes": {
+                name: list(getattr(self, name).shape) for name in self.serialization_buffers
+            },
+            "native_teacher_ids": list(self.native_teacher_ids),
+            "native_teacher_offsets": list(self.native_teacher_offsets),
+        }
+
+    @classmethod
+    def from_config(
+        cls, config: dict, *, attn_implementation: str | None = None,
+    ) -> "CompactNativeClockTalker":
+        """Build the compact modules and shared weights for checkpoint loading."""
+        from transformers import Qwen3Config, Qwen3Model
+
+        model = cls()
+        model.hidden_size = int(config["hidden_size"])
+        model.codebook_size = int(config["codebook_size"])
+        model.eos_class = model.codebook_size
+        backbone_config = Qwen3Config(**config["backbone_config"])
+        backbone_config._attn_implementation = attn_implementation or config["attn_implementation"]
+        model.backbone = Qwen3Model(backbone_config)
+        model.backbone.embed_tokens = None
+        model.text_embedding = ProjectedTextEmbedding(
+            torch.empty(int(config["projected_num_embeddings"]), model.hidden_size),
+            torch.empty(0, dtype=torch.long), int(config["teacher_vocab_size"]),
+        )
+        model.text_projection = nn.Identity()
+        model.codec_history_embeddings = nn.ModuleList(
+            nn.Embedding(model.codebook_size, model.hidden_size)
+            for _ in range(int(config["num_code_groups"]))
+        )
+        model.codec_bos = nn.Parameter(torch.empty(model.hidden_size))
+        model.q0_head = nn.Linear(model.hidden_size, model.codebook_size, bias=False)
+        model.codec_eos_head = nn.Linear(model.hidden_size, 1, bias=False)
+        model.residual_predictor = Qwen3TTSResidualPredictor(
+            model.hidden_size, model.codebook_size, config["residual_predictor_config"],
+        )
+        model.residual_predictor.q0_embedding = model.codec_history_embeddings[0]
+        for group in range(len(model.residual_predictor.code_embeddings)):
+            model.residual_predictor.code_embeddings[group] = model.codec_history_embeddings[group + 1]
+        model.native_teacher_ids = list(config["native_teacher_ids"])
+        model.native_teacher_offsets = list(config["native_teacher_offsets"])
+        for name in cls.serialization_buffers:
+            model.register_buffer(name, torch.empty(config["prefix_shapes"][name]))
+        return model.prepare_for_serialization()
+
+    @classmethod
+    def from_teacher(
+        cls, teacher_path: str | Path, native_tokenizer_path: str | Path,
+        speaker_vector_path: str | Path, *, dtype: torch.dtype = torch.float32,
+        attn_implementation: str = "eager", device: str | torch.device = "cpu",
+        projection_batch_size: int = 1, projected_table_path: str | Path | None = None,
+    ) -> "CompactNativeClockTalker":
+        source = NativeClockTalker.from_teacher(
+            teacher_path, native_tokenizer_path, speaker_vector_path, dtype=dtype,
+            attn_implementation=attn_implementation,
+        ).eval().to(device)
+        return cls.from_native(source, projection_batch_size=projection_batch_size,
+                               projected_table_path=projected_table_path)
+
+    @classmethod
+    @torch.no_grad()
+    def from_native(
+        cls, source: NativeClockTalker, *, projection_batch_size: int = 1,
+        projected_table_path: str | Path | None = None,
+    ) -> "CompactNativeClockTalker":
+        """Reuse acoustic weights; leave the source model's modules unmodified.
+
+        A batch size of one repeats the streaming projection's matrix shape.
+        Larger batches use GEMM and may change floating-point rounding.
+        """
+        model = cls()
+        model.hidden_size = source.hidden_size
+        for name in ("backbone", "codec_history_embeddings", "q0_head", "codec_eos_head"):
+            setattr(model, name, getattr(source, name))
+        model.codec_bos = source.codec_bos
+        model.native_teacher_ids = list(source.native_teacher_ids)
+        model.native_teacher_offsets = list(source.native_teacher_offsets)
+        for name, buffer in source.named_buffers(recurse=False):
+            model.register_buffer(name, buffer, persistent=name not in source._non_persistent_buffers_set)
+
+        device = source.text_embedding.weight.device
+        teacher_ids = torch.tensor(sorted(set(source.native_teacher_ids)), device=device)
+        table_path = Path(projected_table_path) if projected_table_path else None
+        if table_path is not None and table_path.exists():
+            from safetensors.torch import load_file
+            table = load_file(str(table_path), device=str(device))
+            if not torch.equal(table["teacher_ids"], teacher_ids):
+                raise ValueError("Projected table belongs to a different native vocabulary")
+            projected = table["projected"].to(dtype=source.text_embedding.weight.dtype)
+        else:
+            projected = source.q0_head.weight.new_empty((len(teacher_ids), source.hidden_size))
+            for start in range(0, len(teacher_ids), projection_batch_size):
+                ids = teacher_ids[start:start + projection_batch_size].reshape(-1, 1)
+                projected[start:start + len(ids)] = source.text_projection(source.text_embedding(ids))[:, 0]
+            if table_path is not None:
+                from safetensors.torch import save_file
+                table_path.parent.mkdir(parents=True, exist_ok=True)
+                save_file({"teacher_ids": teacher_ids.cpu(), "projected": projected.cpu()}, str(table_path))
+        model.text_embedding = ProjectedTextEmbedding(projected, teacher_ids, source.text_embedding.num_embeddings)
+        model.text_projection = nn.Identity()
+
+        # These fifteen input tables were copied from the same teacher tensors.
+        model.residual_predictor = copy.deepcopy(source.residual_predictor)
+        pairs = [(model.residual_predictor, "q0_embedding", model.codec_history_embeddings[0])]
+        pairs.extend((model.residual_predictor.code_embeddings, str(group), model.codec_history_embeddings[group + 1])
+                     for group in range(len(model.residual_predictor.code_embeddings)))
+        for owner, name, history_embedding in pairs:
+            if not torch.equal(getattr(owner, name).weight, history_embedding.weight):
+                raise ValueError("Codec input embeddings have diverged and cannot be shared")
+            setattr(owner, name, history_embedding)
+        return model.eval()

compaction.json

@@ -0,0 +1,23 @@
+{
+  "source": "/default-filesys/workspace/chenjunzhe/EdgeInstant-1.5b/runs/bilingual_s5_full/final",
+  "output": "/default-filesys/workspace/chenjunzhe/EdgeInstant-1.5b/runs/bilingual_s5_full/final_compact",
+  "precision": "torch.bfloat16",
+  "preserved_modalities": [
+    "text",
+    "audio_input",
+    "image",
+    "video",
+    "speech_output"
+  ],
+  "source_parameters": 2031972995,
+  "exported_parameters": 2031710851,
+  "removed_weights": [
+    "codec.quantizer.rvq_first.input_proj.weight",
+    "codec.quantizer.rvq_rest.input_proj.weight"
+  ],
+  "source_weight_bytes": 5677248858,
+  "exported_weight_bytes": 4115251826,
+  "equivalence_reference": "Source loaded with dtype=auto, as in run_mmau.py; align and audio_special remain FP32.",
+  "cache_implementation": "dynamic",
+  "recurrent_decode": "cuda_graph"
+}

configuration_edgeinstant.py

@@ -0,0 +1,95 @@
+"""Configuration for the AudioIn, Thinker, Talker and speech decoder."""
+from __future__ import annotations
+
+from transformers import PretrainedConfig
+from transformers.models.qwen3_5.configuration_qwen3_5 import Qwen3_5Config
+from transformers.models.qwen3_omni_moe.configuration_qwen3_omni_moe import (
+    Qwen3OmniMoeAudioEncoderConfig,
+)
+
+
+class EdgeInstantConfig(PretrainedConfig):
+    model_type = "edgeinstant"
+    sub_configs = {
+        "thinker_config": Qwen3_5Config,
+        "audio_config": Qwen3OmniMoeAudioEncoderConfig,
+    }
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        thinker_config=None,
+        audio_config=None,
+        projector_config=None,
+        talker_config=None,
+        codec_config=None,
+        audio_tap="proj2",
+        audio_pool=1,
+        audio_stack=1,
+        audio_special_ids=None,
+        audio_pad_id=248076,
+        sampling_rate=16000,
+        output_sampling_rate=24000,
+        max_audio_seconds=None,
+        qa_prompt="Listen to the user's speech and answer.",
+        thinker_special_ids=None,
+        think_start_id=248068,
+        think_end_id=248069,
+        **kwargs,
+    ):
+        self.thinker_config = (
+            thinker_config if isinstance(thinker_config, Qwen3_5Config)
+            else Qwen3_5Config(**(thinker_config or {}))
+        )
+        self.audio_config = (
+            audio_config if isinstance(audio_config, Qwen3OmniMoeAudioEncoderConfig)
+            else Qwen3OmniMoeAudioEncoderConfig(**(audio_config or {}))
+        )
+        self.audio_tap = audio_tap
+        self.audio_pool = int(audio_pool)
+        self.audio_stack = int(audio_stack)
+        if audio_tap not in {"proj2", "ln_post"}:
+            raise ValueError(f"Unsupported AudioIn tower output: {audio_tap}")
+        if self.audio_pool < 1 or self.audio_stack < 1:
+            raise ValueError("audio_pool and audio_stack must be positive integers")
+        audio_size = self.audio_config.d_model if audio_tap == "ln_post" else self.audio_config.output_dim
+        hidden_size = self.thinker_config.text_config.hidden_size
+        self.projector_config = {
+            "norm_size": audio_size,
+            "input_size": audio_size * self.audio_stack,
+            "intermediate_size": None,
+            "output_size": hidden_size,
+            "align_mode": "replace",
+            "audio_out_norm": False,
+            **(projector_config or {}),
+        }
+        self.talker_config = dict(talker_config or {})
+        self.codec_config = dict(codec_config or {})
+        self.audio_special_ids = list(audio_special_ids if audio_special_ids is not None else [248070, 248071, 248076])
+        self.audio_pad_id = int(audio_pad_id)
+        self.sampling_rate = int(sampling_rate)
+        self.output_sampling_rate = int(output_sampling_rate)
+        self.max_audio_seconds = max_audio_seconds
+        self.qa_prompt = qa_prompt
+        self.thinker_special_ids = list(thinker_special_ids or [])
+        self.think_start_id = int(think_start_id)
+        self.think_end_id = int(think_end_id)
+        kwargs.setdefault("tie_word_embeddings", self.thinker_config.tie_word_embeddings)
+        kwargs.setdefault("pad_token_id", self.thinker_config.text_config.pad_token_id)
+        kwargs.setdefault("bos_token_id", self.thinker_config.text_config.bos_token_id)
+        kwargs.setdefault("eos_token_id", self.thinker_config.text_config.eos_token_id)
+        kwargs.setdefault("architectures", ["EdgeInstantForConditionalGeneration"])
+        kwargs.setdefault("processor_class", "EdgeInstantProcessor")
+        kwargs.setdefault("auto_map", {
+            "AutoConfig": "configuration_edgeinstant.EdgeInstantConfig",
+            "AutoModel": "modeling_edgeinstant.EdgeInstantForConditionalGeneration",
+            "AutoModelForCausalLM": "modeling_edgeinstant.EdgeInstantForConditionalGeneration",
+            "AutoProcessor": "processing_edgeinstant.EdgeInstantProcessor",
+        })
+        super().__init__(**kwargs)
+
+    def get_text_config(self, decoder=None, encoder=None):
+        return self.thinker_config.get_text_config(decoder=decoder, encoder=encoder)
+
+
+EdgeInstantConfig.register_for_auto_class()

generation.py

@@ -0,0 +1,22 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from enum import Enum
+
+import torch
+
+
+class AdvanceStatus(str, Enum):
+    WAIT_INPUT = "WAIT_INPUT"
+    PRODUCED = "PRODUCED"
+    END_AUDIO = "END_AUDIO"
+    FORCED_STOP = "FORCED_STOP"
+
+
+@dataclass
+class GenerationResult:
+    frames: list[torch.Tensor]
+    status: AdvanceStatus
+    actions: list[int]
+    consumed_text_tokens: int
+    text_end_consumed: bool

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "cache_implementation": "dynamic",
+  "eos_token_id": [
+    248046,
+    248046
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 248044,
+  "transformers_version": "5.12.1",
+  "use_cache": true
+}

image_processor/preprocessor_config.json

@@ -0,0 +1,26 @@
+{
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "merge_size": 2,
+  "patch_size": 16,
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 16777216,
+    "shortest_edge": 65536
+  },
+  "temporal_patch_size": 2
+}

image_processor/video_preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+  "size": {
+    "longest_edge": 25165824,
+    "shortest_edge": 4096
+  },
+  "patch_size": 16,
+  "temporal_patch_size": 2,
+  "merge_size": 2,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor_type": "Qwen3VLVideoProcessor"
+}

inference_edgeinstant.py

@@ -0,0 +1,65 @@
+"""CUDA graph replay of Qwen3.5's unchanged recurrent decode operations."""
+from __future__ import annotations
+
+import torch
+from transformers.cache_utils import DynamicCache
+from transformers.models.qwen3_5.modeling_qwen3_5 import Qwen3_5DecoderLayer
+
+
+class EdgeInstantDecoderLayer(Qwen3_5DecoderLayer):
+    def __init__(self, config, layer_idx):
+        super().__init__(config, layer_idx)
+        self.config = config
+        self.layer_idx = layer_idx
+        self._decode_graph = None
+
+    def train(self, mode=True):
+        self._decode_graph = None
+        return super().train(mode)
+
+    def _apply(self, fn, recurse=True):
+        self._decode_graph = None
+        return super()._apply(fn, recurse=recurse)
+
+    def forward(self, hidden_states, position_embeddings, attention_mask=None,
+                position_ids=None, past_key_values=None, **kwargs):
+        cache_params = past_key_values
+        if (self.training or torch.is_grad_enabled() or hidden_states.device.type != "cuda"
+                or hidden_states.shape[1] != 1 or cache_params is None
+                or attention_mask is not None or torch.compiler.is_compiling()
+                or not cache_params.has_previous_state(self.layer_idx)):
+            return super().forward(hidden_states, position_embeddings, attention_mask,
+                                   position_ids, past_key_values, **kwargs)
+        layer = cache_params.layers[self.layer_idx]
+        shape = (hidden_states.shape, hidden_states.dtype, hidden_states.device,
+                 layer.conv_states.shape, layer.conv_states.dtype,
+                 layer.recurrent_states.shape, layer.recurrent_states.dtype)
+        state = self._decode_graph
+        if state is None or state[0] != shape:
+            inputs = torch.empty_like(hidden_states)
+            graph_cache = DynamicCache(config=self.config)
+            graph_cache.update_conv_state(layer.conv_states, self.layer_idx)
+            graph_cache.update_recurrent_state(layer.recurrent_states, self.layer_idx)
+            inputs.copy_(hidden_states)
+            current = torch.cuda.current_stream(hidden_states.device)
+            stream = torch.cuda.Stream(device=hidden_states.device)
+            stream.wait_stream(current)
+            with torch.cuda.stream(stream):
+                for _ in range(3):
+                    super().forward(inputs, None, past_key_values=graph_cache)
+            current.wait_stream(stream)
+            graph = torch.cuda.CUDAGraph()
+            with torch.cuda.graph(graph, stream=stream):
+                outputs = super().forward(inputs, None, past_key_values=graph_cache)
+            state = (shape, inputs, graph_cache, graph, outputs)
+            self._decode_graph = state
+        _, inputs, graph_cache, graph, outputs = state
+        # Keep each caller's cache independent, including caches returned by generate.
+        inputs.copy_(hidden_states)
+        graph_cache.update_conv_state(layer.conv_states, self.layer_idx)
+        graph_cache.update_recurrent_state(layer.recurrent_states, self.layer_idx)
+        graph.replay()
+        graph_layer = graph_cache.layers[self.layer_idx]
+        cache_params.update_conv_state(graph_layer.conv_states, self.layer_idx)
+        cache_params.update_recurrent_state(graph_layer.recurrent_states, self.layer_idx)
+        return outputs.clone()

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a659eaf53119fbb56590f3424e96e09dafeb9350d0f8b4633552db0f6d670b1f
+size 1998027410

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ecbfa3ea1abdfddc601ba3f87d041aeed8e78becc56f994cd6954ff560665e4
+size 1996358360

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dd9cb8877ae15c36adabc56b5cbf998333182334213a3dc89372e3b918ae025
+size 120866056

modeling_edgeinstant.py

@@ -0,0 +1,392 @@
+"""Audio-conditioned Thinker and native-token speech generation."""
+from __future__ import annotations
+
+from typing import Iterator
+
+import torch
+from torch import nn
+from torch.nn.attention import SDPBackend, sdpa_kernel
+from transformers import GenerationMixin, PreTrainedModel, Qwen3_5ForConditionalGeneration
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.models.qwen3_omni_moe.modeling_qwen3_omni_moe import (
+    Qwen3OmniMoeAudioEncoder,
+    _get_feat_extract_output_lengths,
+)
+
+from .configuration_edgeinstant import EdgeInstantConfig
+from .codec_edgeinstant import EdgeInstantCodecDecoder
+from .inference_edgeinstant import EdgeInstantDecoderLayer
+from .compact_native_clock import CompactNativeClockTalker
+
+
+class EdgeInstantThinker(Qwen3_5ForConditionalGeneration):
+    """Qwen3.5 with CUDA graph replay for cached recurrent decoding."""
+
+    _no_split_modules = ["Qwen3_5DecoderLayer", "EdgeInstantDecoderLayer", "Qwen3_5VisionBlock"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        for index in range(len(self.model.language_model.layers)):
+            if config.text_config.layer_types[index] == "linear_attention":
+                self.model.language_model.layers[index] = EdgeInstantDecoderLayer(config.text_config, index)
+
+class EdgeInstantAudioProjector(nn.Module):
+    def __init__(self, config: dict, hidden_size: int):
+        super().__init__()
+        self.hidden_size = hidden_size
+        self.mode = config.get("align_mode", "replace")
+        if self.mode == "residual" and int(config["output_size"]) != hidden_size:
+            raise ValueError("Residual audio projection requires audio_split=1 (output_size must equal Thinker hidden_size)")
+        self.out_norm = bool(config.get("audio_out_norm", False))
+        self.norm = nn.LayerNorm(int(config["norm_size"]))
+        middle = config.get("intermediate_size")
+        self.proj = nn.Linear(int(config["input_size"]), int(middle or config["output_size"]))
+        self.proj2 = nn.Linear(int(middle), int(config["output_size"])) if middle else None
+        self.gate = nn.Parameter(torch.zeros(()))
+        self.out_scale = nn.Parameter(torch.ones(()))
+
+    def forward(self, audio: torch.Tensor, pad_embedding: torch.Tensor) -> torch.Tensor:
+        shape = audio.shape
+        frames = audio.float().reshape(*shape[:-1], -1, self.norm.normalized_shape[0])
+        mapped = self.proj(self.norm(frames).reshape(shape))
+        if self.proj2 is not None:
+            mapped = self.proj2(nn.functional.gelu(mapped))
+        if self.mode == "residual":
+            return pad_embedding.to(mapped.dtype) + self.gate.clamp(0, 4) * mapped
+        if self.out_norm:
+            mapped = nn.functional.normalize(mapped, dim=-1, eps=1e-6) * self.hidden_size ** 0.5
+        return (mapped * self.out_scale).reshape(-1, self.hidden_size)
+
+
+class EdgeInstantForConditionalGeneration(PreTrainedModel, GenerationMixin):
+    config_class = EdgeInstantConfig
+    base_model_prefix = "edgeinstant"
+    main_input_name = "input_ids"
+    _supports_sdpa = True
+    _keep_in_fp32_modules_strict = ["align", "audio_special"]
+    _no_split_modules = ["Qwen3OmniMoeAudioEncoder", "EdgeInstantAudioProjector", "EdgeInstantCodecDecoder"]
+
+    def __init__(self, config, *, thinker=None, audio_tower=None, align=None,
+                 talker=None, codec=None, audio_special_delta=None):
+        super().__init__(config)
+        supplied = [component for component in (thinker, audio_tower, align, talker, codec) if component is not None]
+        self.thinker = thinker if thinker is not None else EdgeInstantThinker(config.thinker_config)
+        self.audio_tower = audio_tower if audio_tower is not None else Qwen3OmniMoeAudioEncoder(config.audio_config)
+        if config.audio_tap == "ln_post":
+            self.audio_tower.proj1 = nn.Identity()
+            self.audio_tower.act = nn.Identity()
+            self.audio_tower.proj2 = nn.Identity()
+        self.align = align if align is not None else EdgeInstantAudioProjector(
+            config.projector_config, config.thinker_config.text_config.hidden_size,
+        )
+        self.audio_special = nn.Embedding(len(config.audio_special_ids), config.thinker_config.text_config.hidden_size)
+        if audio_special_delta is not None:
+            self.audio_special.weight = nn.Parameter(audio_special_delta.detach().clone())
+            supplied.append(self.audio_special)
+        self.talker = talker if talker is not None else CompactNativeClockTalker.from_config(config.talker_config)
+        self.talker.prepare_for_serialization()
+        self.codec = codec if codec is not None else (
+            EdgeInstantCodecDecoder(config.codec_config) if config.codec_config else None
+        )
+        self._tied_weights_keys = {
+            f"talker.{target}": f"talker.{source}"
+            for target, source in self.talker.tied_weight_keys.items()
+        }
+        for component in supplied:
+            for module in component.modules():
+                module._is_hf_initialized = True
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.thinker.get_input_embeddings()
+
+    def set_input_embeddings(self, embeddings):
+        self.thinker.set_input_embeddings(embeddings)
+
+    def get_output_embeddings(self):
+        return self.thinker.get_output_embeddings()
+
+    def _base_thinker(self):
+        return self.thinker.get_base_model() if hasattr(self.thinker, "get_base_model") else self.thinker
+
+    def encode_audio(self, input_features, feature_attention_mask=None, tower=None, feature_lengths=None):
+        """Return packed audio encoder features before the trainable projector."""
+        from transformers.models.qwen3_omni_moe.modeling_qwen3_omni_moe import (
+            chunk_and_pad_features, get_audio_cu_seqlens, get_valid_indices,
+        )
+        from transformers.utils.generic import is_flash_attention_requested
+
+        tower = self.audio_tower if tower is None else tower
+        tower_weight = next(tower.parameters())
+        if feature_lengths is not None:
+            lengths = torch.as_tensor(feature_lengths, dtype=torch.long).cpu()
+        elif feature_attention_mask is not None:
+            lengths = feature_attention_mask.sum(-1).to(device="cpu", dtype=torch.long)
+        else:
+            lengths = torch.full((input_features.shape[0],), input_features.shape[-1], dtype=torch.long)
+        if bool((lengths <= 0).any()):
+            raise ValueError("Audio must contain at least one valid mel frame")
+        packed = torch.cat([row[:, :int(length)] for row, length in zip(input_features, lengths)], dim=-1)
+        packed = packed.to(device=tower_weight.device, dtype=tower_weight.dtype)
+        padded_feature, chunk_lengths = chunk_and_pad_features(packed, lengths, tower.n_window)
+        valid_indices = get_valid_indices(chunk_lengths).to(tower_weight.device)
+        cu_seqlens = get_audio_cu_seqlens(chunk_lengths, lengths, tower.n_window_infer, tower.n_window)
+        # SDPA/eager split each attention window in Python; keep their boundaries on the CPU.
+        if is_flash_attention_requested(tower.config):
+            cu_seqlens = cu_seqlens.to(tower_weight.device)
+        hidden = tower(
+            input_features=packed, feature_lens=lengths,
+            padded_feature=padded_feature, chunk_lengths=chunk_lengths,
+            valid_indices=valid_indices, cu_seqlens=cu_seqlens,
+        ).last_hidden_state
+        return self.group_audio_features(hidden, lengths)
+
+    def group_audio_features(self, hidden, frame_lengths):
+        """Apply per-recording frame stacking or pooling to packed encoder output."""
+        if self.config.audio_stack == 1 and self.config.audio_pool == 1:
+            return hidden
+        counts = _get_feat_extract_output_lengths(frame_lengths).tolist()
+        processed = []
+        for row in hidden.split(counts):
+            if self.config.audio_stack > 1:
+                padding = -len(row) % self.config.audio_stack
+                if padding:
+                    row = torch.cat((row, row[-1:].expand(padding, -1)))
+                row = row.reshape(len(row) // self.config.audio_stack, -1)
+            elif self.config.audio_pool > 1:
+                row = torch.stack([chunk.mean(0) for chunk in row.split(self.config.audio_pool)])
+            processed.append(row)
+        return torch.cat(processed)
+
+    def pad_embedding(self):
+        """Audio placeholder embedding including its learned special-token delta."""
+        embed = self.get_input_embeddings()
+        pad = embed.weight[self.config.audio_pad_id]
+        if self.config.audio_pad_id in self.config.audio_special_ids:
+            slot = self.config.audio_special_ids.index(self.config.audio_pad_id)
+            pad = pad + self.audio_special.weight[slot].to(pad)
+        return pad
+
+    def get_audio_features(self, input_features, feature_attention_mask, feature_lengths=None):
+        features = self.encode_audio(input_features, feature_attention_mask, feature_lengths=feature_lengths)
+        device = self.align.proj.weight.device
+        return self.align(features.to(device), self.pad_embedding().to(device))
+
+    def merge_audio_embeds(self, input_ids, input_features, feature_attention_mask, feature_lengths=None):
+        vectors = self.get_audio_features(input_features, feature_attention_mask, feature_lengths)
+        return self.merge_custom_audio(input_ids, vectors)
+
+    def merge_custom_audio(self, input_ids, audio_vectors):
+        """Insert projected or diagnostic audio vectors in the model's audio slots."""
+        embedding = self.get_input_embeddings()
+        ids = input_ids.to(embedding.weight.device)
+        embeds = embedding(ids)
+        for slot, token_id in enumerate(self.config.audio_special_ids):
+            embeds = embeds + (ids == token_id).unsqueeze(-1) * self.audio_special.weight[slot].to(embeds)
+        vectors = audio_vectors.to(embeds)
+        mask = ids == self.config.audio_pad_id
+        if int(mask.sum()) != len(vectors):
+            raise ValueError(f"Audio slots {int(mask.sum())} do not match encoded vectors {len(vectors)}")
+        return embeds.masked_scatter(mask.unsqueeze(-1).expand_as(embeds), vectors)
+
+    def forward(self, input_ids=None, attention_mask=None, input_features=None,
+                feature_attention_mask=None, inputs_embeds=None, labels=None,
+                audio_feature_lengths=None, answer_loss_reduction=None,
+                eos_loss_weight=1.0, unlikelihood_mask=None, unlikelihood_weight=1.0, **kwargs):
+        if input_features is not None:
+            if input_ids is None or feature_attention_mask is None:
+                raise ValueError("Audio forward requires input_ids and feature_attention_mask")
+            cache = kwargs.get("past_key_values")
+            if cache is None or cache.get_seq_length() == 0:
+                self._base_thinker().model.rope_deltas = None
+            inputs_embeds = self.merge_audio_embeds(
+                input_ids, input_features, feature_attention_mask, audio_feature_lengths,
+            )
+            if kwargs.get("position_ids") is None and (
+                kwargs.get("image_grid_thw") is not None or kwargs.get("video_grid_thw") is not None
+            ):
+                kwargs["position_ids"] = self._base_thinker().model.compute_3d_position_ids(
+                    input_ids, inputs_embeds, attention_mask=attention_mask,
+                    **{key: kwargs.get(key) for key in ("image_grid_thw", "video_grid_thw", "mm_token_type_ids")},
+                )
+            input_ids = None
+        if answer_loss_reduction is not None:
+            if labels is None or answer_loss_reduction not in {"token_sum", "example_sum"}:
+                raise ValueError("Answer loss requires labels and token_sum or example_sum reduction")
+            # LoRA lives on the base model's decoder linears; the vocabulary projection
+            # only needs hidden states that predict a supervised answer token or EOS.
+            thinker = self._base_thinker()
+            outputs = thinker.model(input_ids=input_ids, inputs_embeds=inputs_embeds,
+                                    attention_mask=attention_mask, **kwargs)
+            targets = labels[:, 1:]
+            mask = targets.ne(-100)
+            hidden = outputs.last_hidden_state[:, :-1][mask]
+            logits = thinker.lm_head(hidden)
+            losses = nn.functional.cross_entropy(logits.float(), targets[mask], reduction="none")
+            if unlikelihood_mask is not None:
+                negative = unlikelihood_mask[:, 1:][mask].bool()
+                if negative.any():
+                    negative_logits = logits[negative].float()
+                    negative_ids = targets[mask][negative, None]
+                    selected = negative_logits.gather(1, negative_ids).squeeze(1)
+                    alternatives = negative_logits.scatter(1, negative_ids, -torch.inf).logsumexp(dim=1)
+                    losses = losses.index_put((negative,),
+                        nn.functional.softplus(selected - alternatives) * unlikelihood_weight)
+            eos_ids = self.config.eos_token_id
+            if isinstance(eos_ids, int):
+                eos_ids = [eos_ids]
+            is_eos = torch.isin(targets[mask], targets.new_tensor(eos_ids))
+            weights = torch.where(is_eos, eos_loss_weight, 1.0)
+            losses = losses * weights
+            if answer_loss_reduction == "example_sum":
+                example_ids = mask.nonzero(as_tuple=True)[0]
+                counts = weights.new_zeros(targets.shape[0]).scatter_add(0, example_ids, weights)
+                losses = losses / counts[example_ids]
+            return CausalLMOutputWithPast(loss=losses.sum())
+        return self.thinker(input_ids=input_ids, inputs_embeds=inputs_embeds,
+                            attention_mask=attention_mask, labels=labels, **kwargs)
+
+    @torch.no_grad()
+    @sdpa_kernel([SDPBackend.FLASH_ATTENTION, SDPBackend.EFFICIENT_ATTENTION, SDPBackend.MATH])
+    def generate(self, input_ids=None, attention_mask=None, input_features=None,
+                 feature_attention_mask=None, **kwargs):
+        """Generate text token IDs, including the supplied prompt as in HF causal LMs."""
+        self._base_thinker().model.rope_deltas = None
+        if input_features is not None:
+            kwargs["inputs_embeds"] = self.merge_audio_embeds(input_ids, input_features, feature_attention_mask)
+        self.thinker.generation_config = self.generation_config
+        kwargs.setdefault("eos_token_id", self.config.eos_token_id)
+        return self.thinker.generate(input_ids=input_ids, attention_mask=attention_mask, **kwargs)
+
+    generate_text = generate
+
+    @torch.inference_mode()
+    def synthesize(self, token_ids, *, language="chinese", max_frames=1536, do_sample=False,
+                   decode=True, **sampling):
+        """Speak one sequence of native Thinker token IDs."""
+        state = self.talker.new_state(language=language, max_frames=max_frames, do_sample=do_sample, **sampling)
+        for token in torch.as_tensor(token_ids).reshape(-1).tolist():
+            state.push_token(token)
+        state.end_input()
+        while not state.ended:
+            state.advance(max_new_frames=4)
+        codes = torch.stack(state.frames) if state.frames else torch.empty((0, 16), dtype=torch.long)
+        waveform = None
+        if decode:
+            if self.codec is None:
+                raise ValueError("The model package does not contain a codec decoder")
+            waveform = self.codec.decode(codes.unsqueeze(0))[0] if len(codes) else torch.empty(0)
+        return {"audio_codes": codes, "audio": waveform, "sampling_rate": self.config.output_sampling_rate,
+                "stop_reason": state.status.value}
+
+    @torch.inference_mode()
+    def generate_speech(self, input_ids=None, *, language="chinese", max_frames=1536,
+                        audio_do_sample=False, **kwargs):
+        """Generate one reply and its waveform; text generation accepts standard HF options."""
+        if input_ids is None or input_ids.shape[0] != 1:
+            raise ValueError("Speech generation accepts one conversation at a time")
+        sequences = self.generate(input_ids=input_ids, **kwargs)
+        if not isinstance(sequences, torch.Tensor):
+            sequences = sequences.sequences
+        reply = sequences[0, input_ids.shape[1]:]
+        special = set(self.config.thinker_special_ids)
+        public, thinking = [], False
+        for token in reply.tolist():
+            if token == self.config.think_start_id:
+                thinking = True
+            elif token == self.config.think_end_id:
+                thinking = False
+            elif not thinking and token not in special:
+                public.append(token)
+        result = self.synthesize(public, language=language, max_frames=max_frames, do_sample=audio_do_sample)
+        return {"sequences": sequences, "text_token_ids": torch.tensor(public), **result}
+
+    @torch.inference_mode()
+    def stream_generate(self, input_ids, attention_mask=None, input_features=None,
+                        feature_attention_mask=None, *, language="chinese", max_new_tokens=512,
+                        max_frames=1536, first_packet_frames=1, chunk_frames=4,
+                        left_context_frames=50, audio_do_sample=False, cancelled=None) -> Iterator[dict]:
+        """Yield native text tokens and continuous 24 kHz audio chunks for one reply."""
+        if input_ids.shape[0] != 1:
+            raise ValueError("Streaming accepts one conversation at a time")
+        state = self.talker.new_state(language=language, max_frames=max_frames, do_sample=audio_do_sample)
+        self._base_thinker().model.rope_deltas = None
+        inputs_embeds = None
+        if input_features is not None:
+            inputs_embeds = self.merge_audio_embeds(input_ids, input_features, feature_attention_mask)
+        if attention_mask is None:
+            attention_mask = torch.ones_like(input_ids)
+        sequences = input_ids
+        sent = 0
+        special = set(self.config.thinker_special_ids)
+        eos = self.config.eos_token_id
+        eos = {eos} if isinstance(eos, int) else set(eos)
+        thinking = False
+
+        def packet():
+            nonlocal sent
+            if len(state.frames) == sent:
+                return None
+            start = max(0, sent - left_context_frames)
+            codes = torch.stack(state.frames[start:]).unsqueeze(0)
+            pcm = self.codec(codes)[0]
+            pcm = pcm[(sent - start) * self.codec.samples_per_frame:]
+            event = {"type": "audio", "audio": pcm.detach().cpu(), "sampling_rate": self.codec.sample_rate,
+                     "sample_start": sent * self.codec.samples_per_frame,
+                     "audio_codes": torch.stack(state.frames[sent:])}
+            sent = len(state.frames)
+            return event
+
+        if self.codec is None:
+            raise ValueError("The model package does not contain a codec decoder")
+        try:
+            inputs = self.thinker.prepare_inputs_for_generation(
+                sequences, inputs_embeds=inputs_embeds, attention_mask=attention_mask,
+                is_first_iteration=True, use_cache=True, logits_to_keep=1,
+            )
+            output = self.thinker(**inputs)
+            cache = output.past_key_values
+            reached_eos = False
+            for _ in range(max_new_tokens):
+                if cancelled is not None and cancelled.is_set():
+                    return
+                token = output.logits[:, -1].argmax(-1)
+                token_id = int(token.item())
+                if token_id in eos:
+                    reached_eos = True
+                    break
+                if token_id == self.config.think_start_id:
+                    thinking = True
+                elif token_id == self.config.think_end_id:
+                    thinking = False
+                elif not thinking and token_id not in special:
+                    yield {"type": "text", "token_id": token_id}
+                    state.push_token(token_id)
+                    state.advance(max_new_frames=first_packet_frames if sent == 0 else chunk_frames)
+                    event = packet()
+                    if event is not None:
+                        yield event
+                    if state.ended:
+                        break
+                sequences = torch.cat((sequences, token[:, None]), dim=-1)
+                attention_mask = torch.cat((attention_mask, torch.ones_like(token[:, None])), dim=-1)
+                inputs = self.thinker.prepare_inputs_for_generation(
+                    sequences, past_key_values=cache, attention_mask=attention_mask,
+                    is_first_iteration=False, next_sequence_length=1, use_cache=True, logits_to_keep=1,
+                )
+                output = self.thinker(**inputs)
+                cache = output.past_key_values
+            if not state.ended:
+                if not reached_eos:
+                    raise RuntimeError(f"Thinker did not reach EOS within {max_new_tokens} tokens")
+                state.end_input()
+                while not state.ended:
+                    if cancelled is not None and cancelled.is_set():
+                        return
+                    state.advance(max_new_frames=chunk_frames)
+                    event = packet()
+                    if event is not None:
+                        yield event
+            yield {"type": "done", "stop_reason": state.status.value, "samples": sent * self.codec.samples_per_frame}
+        finally:
+            state.cancel()

native_clock_talker.py

@@ -0,0 +1,257 @@
+from __future__ import annotations
+
+from collections import deque
+from dataclasses import dataclass, field
+import json
+from pathlib import Path
+
+import torch
+from safetensors import safe_open
+from torch import nn
+
+from .teacher import TeacherAcousticModel
+from .sequence import Action
+from .sampling import sample_token
+from .generation import AdvanceStatus, GenerationResult
+
+
+class NativeClockTalker(nn.Module):
+    """Teacher-initialized acoustic model with an incremental native text clock."""
+
+    codebook_size = 2048
+    hidden_size = 1024
+    eos_class = 2048
+    supported_languages = ("chinese", "english", "auto")
+
+    @classmethod
+    @torch.no_grad()
+    def from_teacher(
+        cls,
+        teacher_path: str | Path,
+        native_tokenizer_path: str | Path,
+        speaker_vector_path: str | Path,
+        *,
+        dtype: torch.dtype = torch.float32,
+        attn_implementation: str = "eager",
+    ) -> "NativeClockTalker":
+        teacher_path = Path(teacher_path)
+        source = TeacherAcousticModel.from_teacher(
+            teacher_path, native_tokenizer_path, speaker_vector_path,
+            dtype=dtype, attn_implementation=attn_implementation,
+        )
+        model = cls()
+        for name in ("backbone", "text_embedding", "text_projection", "codec_history_embeddings",
+                     "residual_predictor", "q0_head"):
+            setattr(model, name, getattr(source, name))
+        model.codec_bos = source.codec_bos
+        model.native_teacher_ids = source.native_teacher_ids.tolist()
+        model.native_teacher_offsets = source.native_teacher_offsets.tolist()
+        model.codec_eos_head = nn.Linear(model.hidden_size, 1, bias=False, dtype=dtype)
+        config = json.loads((teacher_path / "config.json").read_text())
+        talker = config["talker_config"]
+        vocabulary = json.loads((teacher_path / "vocab.json").read_text())
+
+        def text(ids):
+            return model.text_projection(model.text_embedding(torch.tensor(ids)))
+
+        tts_bos, tts_eos, tts_pad = text([config["tts_bos_token_id"], config["tts_eos_token_id"],
+                                       config["tts_pad_token_id"]])
+        role = text([config["im_start_token_id"], config["assistant_token_id"], vocabulary["Ċ"]])
+        with safe_open(teacher_path / "model.safetensors", framework="pt", device="cpu") as weights:
+            codec = weights.get_tensor("talker.model.codec_embedding.weight").to(dtype=dtype)
+            language = [talker["codec_think_id"], talker["codec_think_bos_id"],
+                        talker["codec_language_id"]["chinese"], talker["codec_think_eos_id"]]
+            prefix = torch.cat((role, codec[language] + tts_pad,
+                                (source.speaker_vector + tts_pad).unsqueeze(0),
+                                (codec[talker["codec_pad_id"]] + tts_bos).unsqueeze(0)), dim=0)
+            english = prefix.clone()
+            english[5] = codec[talker["codec_language_id"]["english"]] + tts_pad
+            auto = torch.cat((role, codec[[talker["codec_nothink_id"],
+                talker["codec_think_bos_id"], talker["codec_think_eos_id"]]] + tts_pad,
+                prefix[7:]), dim=0)
+            model.codec_eos_head.weight.copy_(
+                weights.get_tensor("talker.codec_head.weight")[talker["codec_eos_token_id"]].unsqueeze(0)
+            )
+        model.register_buffer("prefix", prefix.unsqueeze(0), persistent=False)
+        model.register_buffer("prefix_english", english.unsqueeze(0), persistent=False)
+        model.register_buffer("prefix_auto", auto.unsqueeze(0), persistent=False)
+        model.register_buffer("tts_eos", tts_eos.view(1, 1, -1), persistent=False)
+        model.register_buffer("tts_pad", tts_pad.view(1, 1, -1), persistent=False)
+        return model
+
+    def new_state(self, **kwargs) -> "NativeClockState":
+        return NativeClockState(model=self, **kwargs)
+
+    def prefix_for_language(self, language: str) -> torch.Tensor:
+        if language == "chinese":
+            return self.prefix
+        if language == "english":
+            return self.prefix_english
+        if language == "auto":
+            return self.prefix_auto
+        raise ValueError(f"Unsupported speech language: {language}")
+
+    def mapped_teacher_ids(self, native_id: int) -> list[int]:
+        if not 0 <= native_id < len(self.native_teacher_offsets) - 1:
+            return []
+        start, end = self.native_teacher_offsets[native_id:native_id + 2]
+        return self.native_teacher_ids[start:end]
+
+    def history_features(self, codes: torch.Tensor) -> torch.Tensor:
+        return sum(embedding(codes[..., group])
+                   for group, embedding in enumerate(self.codec_history_embeddings))
+
+
+@dataclass
+class NativeClockState:
+    model: NativeClockTalker
+    language: str = "chinese"
+    max_frames: int = 1536
+    max_events: int = 4096
+    do_sample: bool = False
+    top_k: int = 50
+    top_p: float = 1.0
+    temperature: float = 0.9
+    repetition_penalty: float = 1.05
+    sampling_generator: torch.Generator | None = None
+    pending: deque = field(default_factory=deque)
+    native_token_ids: list[int] = field(default_factory=list)
+    native_remaining: list[int] = field(default_factory=list)
+    teacher_token_ids: list[int] = field(default_factory=list)
+    teacher_token_owners: list[int] = field(default_factory=list)
+    consumed_teacher_tokens: int = 0
+    consumed_tokens: int = 0
+    frames: list[torch.Tensor] = field(default_factory=list)
+    actions: list[int] = field(default_factory=list)
+    frame_conditioned_text_tokens: list[int] = field(default_factory=list)
+    frame_arrived_text_tokens: list[int] = field(default_factory=list)
+    frame_available_teacher_tokens: list[int] = field(default_factory=list)
+    frame_consumed_teacher_tokens: list[int] = field(default_factory=list)
+    cache: object | None = None
+    previous_codes: torch.Tensor | None = None
+    input_ended: bool = False
+    text_end_consumed: bool = False
+    text_end_at_codec_frame: int | None = None
+    ended: bool = False
+    cancelled: bool = False
+    event_count: int = 0
+    suppressed_early_eos: int = 0
+    q0_eos_generated: bool = False
+    status: AdvanceStatus = AdvanceStatus.WAIT_INPUT
+
+    @property
+    def consumed_native_tokens(self) -> int:
+        return self.consumed_tokens
+
+    def _update_consumed_native(self) -> None:
+        while self.consumed_tokens < len(self.native_remaining) and self.native_remaining[self.consumed_tokens] == 0:
+            self.consumed_tokens += 1
+
+    def push_token(self, native_id: int) -> None:
+        if self.input_ended or self.ended:
+            raise RuntimeError("Cannot append native text after the input has ended")
+        pieces = self.model.mapped_teacher_ids(int(native_id))
+        index = len(self.native_token_ids)
+        self.native_token_ids.append(int(native_id))
+        self.native_remaining.append(len(pieces))
+        self.teacher_token_ids.extend(pieces)
+        self.teacher_token_owners.extend([index] * len(pieces))
+        self.pending.extend((piece, index) for piece in pieces)
+        self._update_consumed_native()
+
+    def end_input(self) -> None:
+        self.input_ended = True
+
+    finish = end_input
+
+    def cancel(self) -> None:
+        self.cancelled = True
+        self.ended = True
+        self.status = AdvanceStatus.FORCED_STOP
+        self.cache = None
+        self.previous_codes = None
+        self.pending.clear()
+
+    def _result(self, status: AdvanceStatus) -> GenerationResult:
+        self.status = status
+        return GenerationResult(self.frames, status, self.actions, self.consumed_tokens, self.text_end_consumed)
+
+    @torch.inference_mode()
+    def advance(self, max_new_frames: int = 4) -> GenerationResult:
+        if self.ended:
+            return self._result(self.status)
+        produced = 0
+        while produced < max_new_frames:
+            if len(self.frames) >= self.max_frames or self.event_count >= self.max_events:
+                self.ended = True
+                return self._result(AdvanceStatus.FORCED_STOP)
+            if self.pending:
+                teacher_id, native_index = self.pending.popleft()
+                ids = torch.tensor([[teacher_id]], device=self.model.q0_head.weight.device)
+                text = self.model.text_projection(self.model.text_embedding(ids))
+                self.native_remaining[native_index] -= 1
+                self._update_consumed_native()
+                self.consumed_teacher_tokens += 1
+            elif not self.input_ended:
+                return self._result(AdvanceStatus.WAIT_INPUT)
+            elif not self.text_end_consumed:
+                text = self.model.tts_eos
+                self.text_end_consumed = True
+                self.text_end_at_codec_frame = len(self.frames)
+            else:
+                text = self.model.tts_pad
+            if self.cache is None:
+                inputs = torch.cat((self.model.prefix_for_language(self.language),
+                                    self.model.codec_bos.view(1, 1, -1) + text), dim=1)
+            else:
+                inputs = self.model.history_features(self.previous_codes).unsqueeze(1) + text
+            output = self.model.backbone(inputs_embeds=inputs, past_key_values=self.cache, use_cache=True)
+            self.cache = output.past_key_values
+            hidden = output.last_hidden_state[:, -1]
+            logits = torch.cat((self.model.q0_head(hidden), self.model.codec_eos_head(hidden)), dim=-1).float()
+            if self.frames and self.repetition_penalty != 1.0:
+                seen = torch.tensor(list({int(frame[0]) for frame in self.frames}), device=logits.device)
+                scores = logits[:, seen]
+                logits[:, seen] = torch.where(scores < 0, scores * self.repetition_penalty,
+                                               scores / self.repetition_penalty)
+            eos_allowed = self.input_ended and not self.pending and self.text_end_consumed
+            if not eos_allowed:
+                self.suppressed_early_eos += int(logits.argmax(-1).item() == self.model.eos_class)
+                logits[:, self.model.eos_class] = -torch.inf
+            q0 = sample_token(logits, do_sample=self.do_sample, top_k=self.top_k, top_p=self.top_p,
+                              temperature=self.temperature, generator=self.sampling_generator)
+            self.event_count += 1
+            if int(q0.item()) == self.model.eos_class:
+                self.ended = True
+                self.q0_eos_generated = True
+                self.actions.append(int(Action.END))
+                return self._result(AdvanceStatus.END_AUDIO)
+            codes = self.model.residual_predictor.generate(hidden, q0, do_sample=self.do_sample,
+                top_k=self.top_k, top_p=self.top_p, temperature=self.temperature, generator=self.sampling_generator)
+            self.previous_codes = codes
+            self.frames.append(codes[0].detach().cpu())
+            self.actions.append(int(Action.EMIT))
+            self.frame_conditioned_text_tokens.append(self.consumed_tokens)
+            self.frame_arrived_text_tokens.append(len(self.native_token_ids))
+            self.frame_available_teacher_tokens.append(len(self.teacher_token_ids))
+            self.frame_consumed_teacher_tokens.append(self.consumed_teacher_tokens)
+            produced += 1
+        return self._result(AdvanceStatus.PRODUCED)
+
+    def speech_unit_report(self) -> dict:
+        return dict(native_token_ids=self.native_token_ids, native_token_count=len(self.native_token_ids),
+                    teacher_token_ids=self.teacher_token_ids, teacher_token_count=len(self.teacher_token_ids),
+                    teacher_token_owners=self.teacher_token_owners,
+                    consumed_native_tokens=self.consumed_tokens, consumed_teacher_tokens=self.consumed_teacher_tokens)
+
+    def report(self) -> dict:
+        return dict(**self.speech_unit_report(), model_variant="teacher_initialized_native_clock_talker_v1",
+                    language=self.language,
+                    terminal_status=self.status.value, q0_eos_generated=self.q0_eos_generated,
+                    q0_eos_teacher_id=2150, suppressed_early_eos=self.suppressed_early_eos,
+                    text_end_consumed=self.text_end_consumed, text_end_at_codec_frame=self.text_end_at_codec_frame,
+                    frame_arrived_text_tokens=self.frame_arrived_text_tokens,
+                    frame_conditioned_text_tokens=self.frame_conditioned_text_tokens,
+                    frame_available_teacher_tokens=self.frame_available_teacher_tokens,
+                    frame_consumed_teacher_tokens=self.frame_consumed_teacher_tokens,
+                    generated_codec_frames=len(self.frames), cancelled=self.cancelled)

preprocessor_config.json

@@ -0,0 +1,14 @@
+{
+  "chunk_length": 30,
+  "dither": 0.0,
+  "feature_extractor_type": "WhisperFeatureExtractor",
+  "feature_size": 128,
+  "hop_length": 160,
+  "n_fft": 400,
+  "n_samples": 480000,
+  "nb_max_frames": 3000,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "return_attention_mask": true,
+  "sampling_rate": 16000
+}

processing_edgeinstant.py

@@ -0,0 +1,228 @@
+"""Text and 16 kHz speech inputs for EdgeInstant."""
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+from transformers import AutoImageProcessor, AutoTokenizer, ProcessorMixin, WhisperFeatureExtractor
+from transformers.dynamic_module_utils import custom_object_save
+from transformers.feature_extraction_utils import BatchFeature
+
+from .configuration_edgeinstant import EdgeInstantConfig
+
+
+class EdgeInstantProcessor(ProcessorMixin):
+    tokenizer_class = "AutoTokenizer"
+    feature_extractor_class = "WhisperFeatureExtractor"
+    model_input_names = ["input_ids", "attention_mask", "input_features", "feature_attention_mask"]
+
+    def __init__(self, tokenizer, feature_extractor, config, image_processor=None, video_processor_config=None):
+        self.tokenizer = tokenizer
+        self.feature_extractor = feature_extractor
+        self.config = config
+        self.image_processor = image_processor
+        self.video_processor_config = video_processor_config
+        self.chat_template = getattr(tokenizer, "chat_template", None)
+
+    def audio_token_count(self, frame_count):
+        """Count encoder frames, temporal grouping and projector output tokens."""
+        frame_count = int(frame_count)
+        # Qwen3-ASR uses three stride-2 convolutions per 100-frame block.
+        count = (frame_count // 100) * 13 + ((frame_count % 100) + 7) // 8
+        stride = self.config.audio_stack if self.config.audio_stack > 1 else self.config.audio_pool
+        count = (count + stride - 1) // stride
+        projector = self.config.projector_config
+        if projector["align_mode"] != "residual":
+            hidden_size = self.config.thinker_config.text_config.hidden_size
+            count = count * projector["output_size"] // hidden_size
+        return count
+
+    def _audio_batch(self, audio, sampling_rate):
+        if sampling_rate != self.config.sampling_rate:
+            raise ValueError(f"Audio must use {self.config.sampling_rate} Hz; received {sampling_rate} Hz")
+        if hasattr(audio, "detach"):
+            audio = audio.detach().cpu().numpy()
+        if isinstance(audio, np.ndarray):
+            batch = [audio] if audio.ndim == 1 else list(audio)
+        elif isinstance(audio, (list, tuple)):
+            batch = [audio] if audio and np.isscalar(audio[0]) else list(audio)
+        else:
+            raise TypeError("audio must be a mono waveform or a batch of mono waveforms")
+        if not batch:
+            raise ValueError("Audio must be a nonempty mono waveform")
+        result = []
+        for waveform in batch:
+            if hasattr(waveform, "detach"):
+                waveform = waveform.detach().cpu().numpy()
+            waveform = np.asarray(waveform, dtype=np.float32)
+            if waveform.ndim != 1 or not waveform.size:
+                raise ValueError("Audio must be a nonempty mono waveform")
+            result.append(waveform)
+        return result
+
+    def audio_features(self, audio, sampling_rate=16000, return_tensors="np"):
+        """Extract complete waveforms with zero context for their final mel frames."""
+        waveforms = self._audio_batch(audio, sampling_rate)
+        hop = self.feature_extractor.hop_length
+        longest = max(len(waveform) for waveform in waveforms)
+        aligned_samples = ((longest + hop - 1) // hop) * hop
+        original_padding = max(self.feature_extractor.n_samples, aligned_samples)
+        context_padding = ((aligned_samples + self.feature_extractor.n_fft + hop - 1) // hop) * hop
+        # Preserve the original STFT reflection boundary for full-length recordings.
+        padded_samples = min(original_padding, context_padding)
+        return self.feature_extractor(
+            waveforms, sampling_rate=self.config.sampling_rate,
+            padding="max_length", max_length=int(padded_samples), truncation=False,
+            return_attention_mask=True, return_tensors=return_tensors,
+        )
+
+    @staticmethod
+    def assistant_prefix(enable_thinking=False):
+        prefix = "<|im_start|>assistant\n<think>\n"
+        return prefix if enable_thinking else prefix + "\n</think>\n\n"
+
+    @staticmethod
+    def _batch_flags(value, batch_size, name):
+        flags = [value] * batch_size if isinstance(value, bool) else list(value)
+        if len(flags) != batch_size or any(not isinstance(flag, bool) for flag in flags):
+            raise ValueError(f"{name} must be a bool or one bool per input")
+        return flags
+
+    def __call__(
+        self,
+        text=None,
+        audio=None,
+        sampling_rate=16000,
+        task="qa",
+        system_prompt=None,
+        enable_thinking=False,
+        audio_first=True,
+        return_tensors="pt",
+        padding=True,
+        **kwargs,
+    ):
+        """Build a generation prompt for each text or speech input.
+
+        A string prompt is shared by all waveforms; a list supplies one prompt
+        per waveform. ``enable_thinking`` accepts a bool or one bool per input.
+        ``audio_first`` controls whether audio precedes the user text, with the
+        same scalar or per-input format.
+        Text padding defaults to the left for batched generation;
+        pass ``padding_side="right"`` for training batches.
+        """
+        if task not in {"qa", "asr"}:
+            raise ValueError(f"Unsupported audio task: {task}")
+        if text is None and audio is None:
+            raise ValueError("Provide text or audio")
+        audio_features = {}
+        if audio is not None:
+            waveforms = self._audio_batch(audio, sampling_rate)
+            features = self.audio_features(waveforms, sampling_rate=sampling_rate)
+            audio_features = {
+                "input_features": features["input_features"],
+                "feature_attention_mask": features["attention_mask"],
+            }
+            counts = [self.audio_token_count(length) for length in features["attention_mask"].sum(-1)]
+            prompts = [text or ""] * len(waveforms) if text is None or isinstance(text, str) else list(text)
+            if len(prompts) != len(waveforms):
+                raise ValueError("Provide one text prompt per waveform, or one shared string prompt")
+            thinking_flags = self._batch_flags(enable_thinking, len(prompts), "enable_thinking")
+            audio_first_flags = self._batch_flags(audio_first, len(prompts), "audio_first")
+            rendered = []
+            for prompt, count, thinking, first in zip(prompts, counts, thinking_flags, audio_first_flags):
+                user_prompt = prompt.strip() or ("Transcribe the speech." if task == "asr" else self.config.qa_prompt)
+                prefix = f"<|im_start|>system\n{system_prompt}<|im_end|>\n" if system_prompt and task != "asr" else ""
+                audio_prompt = f"<|audio_start|>{'<|audio_pad|>' * count}<|audio_end|>"
+                user_content = f"{audio_prompt}\n{user_prompt}" if first else f"{user_prompt}\n{audio_prompt}"
+                rendered.append(
+                    f"{prefix}<|im_start|>user\n{user_content}<|im_end|>\n{self.assistant_prefix(thinking)}"
+                )
+        else:
+            prompts = [text] if isinstance(text, str) else list(text)
+            thinking_flags = self._batch_flags(enable_thinking, len(prompts), "enable_thinking")
+            rendered = []
+            for prompt, thinking in zip(prompts, thinking_flags):
+                messages = []
+                if system_prompt:
+                    messages.append({"role": "system", "content": system_prompt})
+                messages.append({"role": "user", "content": prompt})
+                rendered.append(self.tokenizer.apply_chat_template(
+                    messages, tokenize=False, add_generation_prompt=True, enable_thinking=thinking,
+                ))
+        kwargs.setdefault("padding_side", "left")
+        kwargs.setdefault("return_token_type_ids", False)
+        tokens = self.tokenizer(
+            rendered, add_special_tokens=False, padding=padding,
+            return_tensors=return_tensors, **kwargs,
+        )
+        return BatchFeature(data={**tokens, **audio_features}, tensor_type=return_tensors)
+
+    def decode(self, *args, **kwargs):
+        return self.tokenizer.decode(*args, **kwargs)
+
+    def batch_decode(self, *args, **kwargs):
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def to_dict(self):
+        return {
+            "processor_class": self.__class__.__name__,
+            "auto_map": {"AutoProcessor": "processing_edgeinstant.EdgeInstantProcessor"},
+            "image_processor_subfolder": "image_processor" if self.image_processor is not None else None,
+        }
+
+    def save_pretrained(self, save_directory, **kwargs):
+        directory = Path(save_directory)
+        directory.mkdir(parents=True, exist_ok=True)
+        self.config.save_pretrained(directory)
+        self.tokenizer.save_pretrained(directory, **kwargs)
+        self.feature_extractor.save_pretrained(directory)
+        if self.image_processor is not None:
+            self.image_processor.save_pretrained(directory / "image_processor")
+            if self.video_processor_config is not None:
+                (directory / "image_processor" / "video_preprocessor_config.json").write_text(
+                    json.dumps(self.video_processor_config, indent=2) + "\n", encoding="utf-8",
+                )
+        processor_file = directory / "processor_config.json"
+        processor_file.write_text(json.dumps(self.to_dict(), indent=2) + "\n", encoding="utf-8")
+        custom_object_save(self, directory)
+        return [str(processor_file)]
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
+        trust_remote_code = kwargs.pop("trust_remote_code", False)
+        kwargs.pop("_from_auto", None)
+        config = kwargs.pop("config", None)
+        hub_keys = {
+            "cache_dir", "force_download", "local_files_only", "token",
+            "revision", "subfolder", "proxies",
+        }
+        hub_kwargs = {key: value for key, value in kwargs.items() if key in hub_keys}
+        if config is None:
+            config = EdgeInstantConfig.from_pretrained(pretrained_model_name_or_path, **hub_kwargs)
+        tokenizer = AutoTokenizer.from_pretrained(
+            pretrained_model_name_or_path, config=config.thinker_config,
+            trust_remote_code=trust_remote_code, **kwargs,
+        )
+        feature_extractor = WhisperFeatureExtractor.from_pretrained(pretrained_model_name_or_path, **hub_kwargs)
+        metadata_path = Path(pretrained_model_name_or_path) / "processor_config.json"
+        if not metadata_path.is_file():
+            from transformers.utils.hub import cached_file
+            metadata_path = Path(cached_file(pretrained_model_name_or_path, "processor_config.json", **hub_kwargs))
+        metadata = json.loads(metadata_path.read_text())
+        image_processor = None
+        video_processor_config = None
+        if metadata.get("image_processor_subfolder"):
+            subfolder = str(Path(hub_kwargs.get("subfolder", "")) / metadata["image_processor_subfolder"])
+            image_kwargs = {**hub_kwargs, "subfolder": subfolder}
+            image_processor = AutoImageProcessor.from_pretrained(pretrained_model_name_or_path, **image_kwargs)
+            from transformers.utils.hub import cached_file
+            video_path = cached_file(pretrained_model_name_or_path, "video_preprocessor_config.json",
+                                     _raise_exceptions_for_missing_entries=False, **image_kwargs)
+            if video_path is not None:
+                video_processor_config = json.loads(Path(video_path).read_text())
+        return cls(tokenizer=tokenizer, feature_extractor=feature_extractor, config=config,
+                   image_processor=image_processor, video_processor_config=video_processor_config)
+
+
+EdgeInstantProcessor.register_for_auto_class()

processor_config.json

@@ -0,0 +1,7 @@
+{
+  "processor_class": "EdgeInstantProcessor",
+  "auto_map": {
+    "AutoProcessor": "processing_edgeinstant.EdgeInstantProcessor"
+  },
+  "image_processor_subfolder": "image_processor"
+}

requirements.txt

@@ -0,0 +1,7 @@
+torch>=2.6
+transformers==5.12.1
+accelerate>=1.1.0
+safetensors
+numpy
+soundfile
+Pillow>=10

residual_predictor.py

@@ -0,0 +1,162 @@
+from __future__ import annotations
+
+from typing import Any
+
+import torch
+from torch import nn
+
+from .sampling import sample_token
+
+
+class Qwen3TTSResidualPredictor(nn.Module):
+    """The complete pretrained five-layer Qwen3-TTS code predictor."""
+
+    def __init__(self, hidden_size: int, codebook_size: int, config: dict[str, Any]):
+        super().__init__()
+        from transformers import Qwen3Config, Qwen3Model
+
+        if hidden_size != int(config["hidden_size"]) or codebook_size != int(
+            config["vocab_size"]
+        ):
+            raise ValueError("Qwen3-TTS residual predictor dimensions do not match")
+        backbone_config = Qwen3Config(
+            vocab_size=codebook_size,
+            hidden_size=hidden_size,
+            intermediate_size=int(config["intermediate_size"]),
+            num_hidden_layers=int(config["num_hidden_layers"]),
+            num_attention_heads=int(config["num_attention_heads"]),
+            num_key_value_heads=int(config["num_key_value_heads"]),
+            head_dim=int(config["head_dim"]),
+            hidden_act=config["hidden_act"],
+            max_position_embeddings=int(config["max_position_embeddings"]),
+            initializer_range=float(config["initializer_range"]),
+            rms_norm_eps=float(config["rms_norm_eps"]),
+            use_cache=True,
+            tie_word_embeddings=False,
+            rope_theta=float(config["rope_theta"]),
+            attention_bias=bool(config["attention_bias"]),
+            attention_dropout=float(config["attention_dropout"]),
+        )
+        backbone_config._attn_implementation = "eager"
+        self.backbone = Qwen3Model(backbone_config)
+        self.backbone.embed_tokens = None
+        self.q0_embedding = nn.Embedding(codebook_size, hidden_size)
+        self.code_embeddings = nn.ModuleList(
+            [nn.Embedding(codebook_size, hidden_size) for _ in range(14)]
+        )
+        self.heads = nn.ModuleList(
+            [nn.Linear(hidden_size, codebook_size, bias=False) for _ in range(15)]
+        )
+        self._greedy_graph = None
+        self._sampling_graph = None
+
+    @torch.no_grad()
+    def prepare_cuda_graph(self) -> None:
+        """Capture single-session greedy inference after loading the checkpoint."""
+        self._greedy_graph = self._capture_generation_graph(do_sample=False)
+
+    @torch.no_grad()
+    def prepare_sampling_cuda_graph(self) -> None:
+        """Capture top-k 50, top-p 1, temperature 0.9 with the default CUDA RNG."""
+        self._sampling_graph = self._capture_generation_graph(do_sample=True)
+
+    def _capture_generation_graph(self, *, do_sample: bool) -> tuple:
+        weight = self.q0_embedding.weight
+        if self.training or weight.device.type != "cuda":
+            raise ValueError("Residual CUDA graph requires a CUDA model in eval mode")
+        with torch.cuda.device(weight.device):
+            hidden = weight.new_zeros((1, weight.shape[1]))
+            q0 = torch.zeros(1, device=weight.device, dtype=torch.long)
+            masks = [weight.new_zeros((1, 1, 2, 2))]
+            masks[0][0, 0, 0, 1] = torch.finfo(weight.dtype).min
+            masks.extend(weight.new_zeros((1, 1, 1, n)) for n in range(3, 17))
+            stream = torch.cuda.Stream(device=weight.device)
+            stream.wait_stream(torch.cuda.current_stream())
+            with torch.cuda.stream(stream):
+                for _ in range(3):
+                    self._generate(hidden, q0, do_sample=do_sample, attention_masks=masks)
+            torch.cuda.current_stream().wait_stream(stream)
+            graph = torch.cuda.CUDAGraph()
+            with torch.cuda.graph(graph):
+                output = self._generate(hidden, q0, do_sample=do_sample, attention_masks=masks)
+            # Keep all captured inputs alive for subsequent replays.
+            return graph, hidden, q0, output, masks
+
+    def forward(self, hidden: torch.Tensor, frame_prefix: torch.Tensor) -> torch.Tensor:
+        if frame_prefix.ndim != 2 or frame_prefix.shape[1] != 16:
+            raise ValueError("frame_prefix must be [N, 16]")
+        inputs = [hidden.unsqueeze(1), self.q0_embedding(frame_prefix[:, :1])]
+        inputs.extend(
+            embedding(frame_prefix[:, group : group + 1])
+            for group, embedding in enumerate(self.code_embeddings, start=1)
+        )
+        outputs = self.backbone(
+            inputs_embeds=torch.cat(inputs, dim=1), use_cache=False
+        ).last_hidden_state
+        return torch.stack(
+            [head(outputs[:, group + 1]) for group, head in enumerate(self.heads)],
+            dim=1,
+        )
+
+    @torch.no_grad()
+    def generate(
+        self,
+        hidden: torch.Tensor,
+        q0: torch.Tensor,
+        *,
+        do_sample: bool = False,
+        top_k: int = 50,
+        top_p: float = 1.0,
+        temperature: float = 0.9,
+        generator: torch.Generator | None = None,
+    ) -> torch.Tensor:
+        captured = self._greedy_graph if not do_sample else (
+            self._sampling_graph
+            if generator is None and top_k == 50 and top_p == 1.0 and temperature == 0.9
+            else None
+        )
+        if captured is not None and hidden.shape[0] == 1:
+            graph, static_hidden, static_q0, output, _ = captured
+            static_hidden.copy_(hidden)
+            static_q0.copy_(q0)
+            graph.replay()
+            return output.clone()
+        return self._generate(
+            hidden, q0, do_sample=do_sample, top_k=top_k, top_p=top_p,
+            temperature=temperature, generator=generator,
+        )
+
+    def _generate(
+        self, hidden: torch.Tensor, q0: torch.Tensor, *,
+        do_sample: bool = False, top_k: int = 50, top_p: float = 1.0,
+        temperature: float = 0.9, generator: torch.Generator | None = None,
+        attention_masks: list[torch.Tensor] | None = None,
+    ) -> torch.Tensor:
+        inputs = torch.cat(
+            [hidden.unsqueeze(1), self.q0_embedding(q0).unsqueeze(1)], dim=1
+        )
+        output = self.backbone(
+            inputs_embeds=inputs, use_cache=True,
+            attention_mask={"full_attention": attention_masks[0]} if attention_masks else None,
+        )
+        cache = output.past_key_values
+        codes = [q0]
+        for group, head in enumerate(self.heads):
+            code = sample_token(
+                head(output.last_hidden_state[:, -1]),
+                do_sample=do_sample,
+                top_k=top_k,
+                top_p=top_p,
+                temperature=temperature,
+                generator=generator,
+            )
+            codes.append(code)
+            if group < len(self.code_embeddings):
+                output = self.backbone(
+                    inputs_embeds=self.code_embeddings[group](code).unsqueeze(1),
+                    past_key_values=cache,
+                    use_cache=True,
+                    attention_mask={"full_attention": attention_masks[group + 1]} if attention_masks else None,
+                )
+                cache = output.past_key_values
+        return torch.stack(codes, dim=-1)

sampling.py

@@ -0,0 +1,32 @@
+from __future__ import annotations
+
+import torch
+
+
+def sample_token(
+    logits: torch.Tensor,
+    *,
+    do_sample: bool = False,
+    top_k: int = 50,
+    top_p: float = 1.0,
+    temperature: float = 0.9,
+    generator: torch.Generator | None = None,
+) -> torch.Tensor:
+    if not do_sample:
+        return logits.argmax(dim=-1)
+    scores = logits.float() / temperature
+    if 0 < top_k < scores.shape[-1]:
+        threshold = scores.topk(top_k, dim=-1).values[..., -1, None]
+        scores = scores.masked_fill(scores < threshold, -torch.inf)
+    if top_p < 1.0:
+        sorted_scores, sorted_indices = scores.sort(dim=-1, descending=True)
+        remove = sorted_scores.softmax(dim=-1).cumsum(dim=-1) > top_p
+        remove[..., 1:] = remove[..., :-1].clone()
+        remove[..., 0] = False
+        sorted_scores = sorted_scores.masked_fill(remove, -torch.inf)
+        scores = torch.full_like(scores, -torch.inf).scatter(
+            -1, sorted_indices, sorted_scores
+        )
+    return torch.multinomial(
+        scores.softmax(dim=-1), num_samples=1, generator=generator
+    ).squeeze(-1)

sequence.py

@@ -0,0 +1,8 @@
+"""Actions reported by native-token speech generation."""
+from enum import IntEnum
+
+
+class Action(IntEnum):
+    READ = 0
+    EMIT = 1
+    END = 2

teacher.py

@@ -0,0 +1,171 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+import torch
+from safetensors import safe_open
+from torch import nn
+import torch.nn.functional as F
+
+from .residual_predictor import Qwen3TTSResidualPredictor
+
+
+def native_teacher_mapping(
+    native_tokenizer_path: str | Path,
+    teacher_path: str | Path,
+    native_vocabulary_size: int | None = None,
+) -> tuple[torch.Tensor, torch.Tensor]:
+    """Map exact ByteLevel pieces to teacher IDs, with empty special-token bags."""
+    native_path = Path(native_tokenizer_path)
+    if native_path.is_dir():
+        native_path = native_path / "tokenizer.json"
+    native = json.loads(native_path.read_text())["model"]["vocab"]
+    teacher = json.loads((Path(teacher_path) / "vocab.json").read_text())
+    visible = list(range(33, 127)) + list(range(161, 173)) + list(range(174, 256))
+    inverse = {chr(byte): byte for byte in visible}
+    inverse.update({chr(256 + index): byte for index, byte in
+                    enumerate(byte for byte in range(256) if byte not in visible)})
+
+    def raw(piece: str) -> bytes:
+        return bytes(inverse[character] for character in piece)
+
+    teacher_bytes = {raw(piece): index for piece, index in teacher.items()}
+    by_first_byte: dict[int, dict[bytes, int]] = {}
+    for piece, index in teacher_bytes.items():
+        by_first_byte.setdefault(piece[0], {})[piece] = index
+    widths = {first: sorted({len(piece) for piece in pieces}, reverse=True)
+              for first, pieces in by_first_byte.items()}
+    native_by_id = {index: raw(piece) for piece, index in native.items()}
+    size = native_vocabulary_size or (max(native_by_id) + 1)
+    flat, offsets = [], [0]
+    for native_id in range(size):
+        piece = native_by_id.get(native_id, b"")
+        if piece in teacher_bytes:
+            flat.append(teacher_bytes[piece])
+        else:
+            position = 0
+            while position < len(piece):
+                candidates = by_first_byte[piece[position]]
+                for width in widths[piece[position]]:
+                    if width > len(piece) - position:
+                        continue
+                    candidate = piece[position:position + width]
+                    if candidate in candidates:
+                        flat.append(candidates[candidate])
+                        position += width
+                        break
+                else:
+                    raise ValueError(f"Teacher vocabulary cannot represent native token {native_id}")
+        offsets.append(len(flat))
+    return torch.tensor(flat, dtype=torch.long), torch.tensor(offsets, dtype=torch.long)
+
+
+class TeacherTextProjection(nn.Module):
+    def __init__(self, text_hidden_size: int, hidden_size: int):
+        super().__init__()
+        self.linear_fc1 = nn.Linear(text_hidden_size, text_hidden_size)
+        self.linear_fc2 = nn.Linear(text_hidden_size, hidden_size)
+
+    def forward(self, hidden: torch.Tensor) -> torch.Tensor:
+        return self.linear_fc2(F.silu(self.linear_fc1(hidden)))
+
+
+class TeacherAcousticModel(nn.Module):
+    """Pretrained text, acoustic and residual modules shared by speech models."""
+
+    codebook_size = 2048
+    num_code_groups = 16
+
+    def __init__(
+        self,
+        config: dict,
+        native_teacher_ids: torch.Tensor,
+        native_teacher_offsets: torch.Tensor,
+        speaker_vector: torch.Tensor,
+        *,
+        attn_implementation: str = "eager",
+    ):
+        super().__init__()
+        from transformers import Qwen3Config, Qwen3Model
+
+        self.config = dict(config)
+        self.hidden_size = int(config["hidden_size"])
+        backbone_config = Qwen3Config(
+            vocab_size=self.codebook_size,
+            hidden_size=self.hidden_size,
+            intermediate_size=int(config["intermediate_size"]),
+            num_hidden_layers=int(config["num_hidden_layers"]),
+            num_attention_heads=int(config["num_attention_heads"]),
+            num_key_value_heads=int(config["num_key_value_heads"]),
+            head_dim=int(config["head_dim"]),
+            hidden_act=config["hidden_act"],
+            max_position_embeddings=int(config["max_position_embeddings"]),
+            rms_norm_eps=float(config["rms_norm_eps"]),
+            rope_theta=float(config["rope_theta"]),
+            attention_bias=bool(config["attention_bias"]),
+            attention_dropout=float(config.get("attention_dropout", 0.0)),
+            tie_word_embeddings=False,
+            use_cache=True,
+        )
+        backbone_config._attn_implementation = attn_implementation
+        self.backbone = Qwen3Model(backbone_config)
+        self.backbone.embed_tokens = None
+        self.text_embedding = nn.Embedding(int(config["text_vocab_size"]), int(config["text_hidden_size"]))
+        self.text_projection = TeacherTextProjection(int(config["text_hidden_size"]), self.hidden_size)
+        self.text_embedding.requires_grad_(False)
+        self.text_projection.requires_grad_(False)
+        self.register_buffer("native_teacher_ids", native_teacher_ids, persistent=False)
+        self.register_buffer("native_teacher_offsets", native_teacher_offsets, persistent=False)
+        self.register_buffer("speaker_vector", speaker_vector.reshape(self.hidden_size), persistent=False)
+        self.codec_history_embeddings = nn.ModuleList(
+            [nn.Embedding(self.codebook_size, self.hidden_size) for _ in range(self.num_code_groups)]
+        )
+        self.codec_bos = nn.Parameter(torch.zeros(self.hidden_size))
+        self.q0_head = nn.Linear(self.hidden_size, self.codebook_size, bias=False)
+        self.residual_predictor = Qwen3TTSResidualPredictor(
+            self.hidden_size, self.codebook_size, config["code_predictor_config"]
+        )
+
+    @classmethod
+    def from_teacher(
+        cls,
+        teacher_path: str | Path,
+        native_tokenizer_path: str | Path,
+        speaker_vector_path: str | Path,
+        *,
+        dtype: torch.dtype = torch.float32,
+        attn_implementation: str = "eager",
+    ) -> "TeacherAcousticModel":
+        teacher_path, native_tokenizer_path = Path(teacher_path), Path(native_tokenizer_path)
+        config = json.loads((teacher_path / "config.json").read_text())["talker_config"]
+        native_root = native_tokenizer_path if native_tokenizer_path.is_dir() else native_tokenizer_path.parent
+        native_config = json.loads((native_root / "config.json").read_text())
+        native_size = int(native_config.get("text_config", native_config)["vocab_size"])
+        ids, offsets = native_teacher_mapping(native_tokenizer_path, teacher_path, native_size)
+        speaker = torch.as_tensor(np.load(speaker_vector_path), dtype=torch.float32)
+        model = cls(config, ids, offsets, speaker, attn_implementation=attn_implementation).to(dtype=dtype)
+        with safe_open(teacher_path / "model.safetensors", framework="pt", device="cpu") as weights:
+            with torch.no_grad():
+                for name, parameter in model.backbone.named_parameters():
+                    parameter.copy_(weights.get_tensor(f"talker.model.{name}"))
+                model.text_embedding.weight.copy_(weights.get_tensor("talker.model.text_embedding.weight"))
+                for name, parameter in model.text_projection.named_parameters():
+                    parameter.copy_(weights.get_tensor(f"talker.text_projection.{name}"))
+                codec = weights.get_tensor("talker.model.codec_embedding.weight")
+                model.codec_history_embeddings[0].weight.copy_(codec[:model.codebook_size])
+                model.codec_bos.copy_(codec[int(config["codec_bos_id"])])
+                model.q0_head.weight.copy_(weights.get_tensor("talker.codec_head.weight")[:model.codebook_size])
+                model.residual_predictor.q0_embedding.weight.copy_(codec[:model.codebook_size])
+                for name, parameter in model.residual_predictor.backbone.named_parameters():
+                    parameter.copy_(weights.get_tensor(f"talker.code_predictor.model.{name}"))
+                for group in range(15):
+                    embedding = weights.get_tensor(f"talker.code_predictor.model.codec_embedding.{group}.weight")
+                    model.codec_history_embeddings[group + 1].weight.copy_(embedding)
+                    if group < 14:
+                        model.residual_predictor.code_embeddings[group].weight.copy_(embedding)
+                    model.residual_predictor.heads[group].weight.copy_(
+                        weights.get_tensor(f"talker.code_predictor.lm_head.{group}.weight")
+                    )
+        return model

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f32ce20dc35f57a7f9ad1eac03525bd7d30f9df8cea6507e958279cc3657706
+size 19989492

tokenizer_config.json

@@ -0,0 +1,36 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": true,
+  "max_length": null,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_to_multiple_of": null,
+  "pad_token": "<|endoftext|>",
+  "pad_token_type_id": 0,
+  "padding_side": "left",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}