Super-squash branch 'main' using huggingface_hub

Co-authored-by: kumass2020 <kumass2020@users.noreply.huggingface.co>
Co-authored-by: ddwkim <ddwkim@users.noreply.huggingface.co>
Co-authored-by: ubuntu <ubuntu@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,5 @@
+model-00001-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00002-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00003-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00004-of-00004.safetensors filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,206 @@
+---
+library_name: transformers
+tags:
+- speech
+- audio
+- multimodal
+license: cc-by-nc-4.0
+language:
+- en
+- ko
+---
+
+# Raon-Speech-9B
+
+<div align="center">
+  <img class="block dark:hidden" src="assets/Raon-Speech-Gradient-Black.png" alt="Raon-Speech Logo" width="400">
+  <img class="hidden dark:block" src="assets/Raon-Speech-Gradient-White.png" alt="Raon-Speech Logo" width="400">
+</div>
+
+<p align="center">
+  <a href="https://www.krafton.ai/ko/"><img src="https://img.shields.io/badge/Homepage-KRAFTON%20AI-blue?style=flat&logo=google-chrome&logoColor=white" alt="Homepage"></a>
+  <a href="https://github.com/krafton-ai/Raon-Speech"><img src="https://img.shields.io/badge/GitHub-Raon-white?style=flat&logo=github&logoColor=black" alt="GitHub"></a>
+  <br>
+  <a href="https://huggingface.co/KRAFTON"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-KRAFTON-yellow?style=flat" alt="Hugging Face"></a>
+  <a href="https://x.com/Krafton_AI"><img src="https://img.shields.io/badge/X-KRAFTON%20AI-white?style=flat&logo=x&logoColor=black" alt="X"></a>
+  <br>
+  <a href="https://creativecommons.org/licenses/by-nc/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey?style=flat" alt="License"></a>
+</p>
+
+<p align="center">
+  Technical Report | Blog (Coming soon)
+  <!-- 📄 <a href="https://arxiv.org/abs/YOUR_PAPER_ID">Technical Report</a> | 📝 <a href="https://YOUR_BLOG_URL">Blog</a> -->
+</p>
+
+RAON-Speech is a 9B-parameter speech language model that supports state-of-the-art speech understanding, answering and generation in English and Korean.
+This model successfully transforms a pre-trained LLM into a SpeechLM to both understand and generate speech without compromising its original language capabilities.
+It trains on millions of hours of English-Korean speech-text datasets with the following training stages: (1) speech encoder-decoder alignment, (2) end-to-end SpeechLM pre-training, and (3) multi-reward DPO-based post-training.
+
+## Key Features
+
+- **End-to-End Speech Language Model**: 9B-parameter multimodal model built on Qwen3 (36 layers, 4096 hidden dim), Qwen3OmniMoeAudioEncoder (24 layers), Mimi codec (32 quantizers), and ECAPA-TDNN speaker encoder.
+- **Bilingual Support**: State-of-the-art speech understanding, answering, and generation in both English and Korean.
+- **Multi-Task Capabilities**: Supports STT (audio → text), TTS (text → audio), TextQA (text + audio → text), and SpeechChat (audio → text) in a single unified model.
+- **Speaker Voice Conditioning**: TTS with optional speaker reference audio for voice cloning via ECAPA-TDNN embeddings.
+- **TTS Continuation**: Generate speech that naturally continues from a reference audio, with prefill-based continuation for seamless prosody.
+- **Multi-Reward DPO Post-Training**: Three-stage training pipeline — (1) speech encoder-decoder alignment, (2) end-to-end SpeechLM pre-training, and (3) multi-reward DPO-based post-training — for high-quality speech generation.
+- **HuggingFace Transformers Integration**: Load and run directly via `AutoModel.from_pretrained` with `trust_remote_code=True` — no custom package installation required.
+
+## Benchmark Results
+
+Measured with LibriSpeech test-clean samples on single-GPU setups via streaming TTS. All values are averaged.
+
+| Metric | RTX 6000 Pro | L40S |
+|--------|-------------|------|
+| **RTF** | 0.27 (3.7× real-time) | 0.45 (2.2× real-time) |
+| **TTFT** | 617 ms | 887 ms |
+| **TBT** | 135 ms | 233 ms |
+
+- **RTF** (Real-Time Factor): Lower is faster. Values below 1.0 mean faster-than-real-time synthesis.
+- **TTFT** (Time to First Token): Latency until the first audio chunk is returned.
+- **TBT** (Time Between Tokens): Average interval between consecutive audio chunks.
+
+
+## Requirements
+
+```bash
+pip install transformers>=4.57.1 torch torchaudio soundfile accelerate
+
+# Optional
+pip install speechbrain  # for TTS with speaker voice conditioning
+pip install gradio       # for Gradio demo
+```
+
+## Quick Start
+
+### Option 1: Load from Hub (recommended)
+
+No `pip install raon` needed.
+
+```python
+from transformers import AutoConfig
+from transformers.dynamic_module_utils import get_class_from_dynamic_module
+
+MODEL_ID = "KRAFTON/Raon-Speech-9B"
+
+_cfg = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=True)
+RaonPipeline = get_class_from_dynamic_module(
+    "modeling_raon.RaonPipeline",
+    MODEL_ID,
+    revision=getattr(_cfg, "_commit_hash", None),
+)
+del _cfg
+
+pipe = RaonPipeline(MODEL_ID, device="cuda", dtype="bfloat16")
+```
+
+### Option 2: With raon package installed
+
+```bash
+git clone https://github.com/krafton-ai/Raon-Speech.git
+cd Raon-Speech/raon
+pip install -e .  # or: uv sync
+```
+
+```python
+from raon import RaonPipeline
+
+# From Hub (local code + Hub weights)
+pipe = RaonPipeline("KRAFTON/Raon-Speech-9B")
+
+# From local path
+pipe = RaonPipeline("/path/to/raon-model")
+```
+
+## Tasks
+
+#### STT (Audio → Text)
+```python
+text = pipe.stt("audio.wav")
+```
+
+#### TTS (Text → Audio)
+```python
+# Without speaker conditioning
+audio, sr = pipe.tts("Hello, how are you?")
+pipe.save_audio((audio, sr), "output.wav")
+
+# With speaker conditioning (requires speechbrain)
+audio, sr = pipe.tts("Hello, how are you?", speaker_audio="speaker_ref.wav")
+```
+
+#### TextQA (Text + Audio → Text)
+```python
+answer = pipe.textqa("What is the speaker saying?", audio="audio.wav")
+```
+
+#### SpeechChat (Audio → Text)
+```python
+answer = pipe.speech_chat("question.wav")
+```
+
+#### Chat (Multimodal)
+```python
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "audio", "audio": "audio.wav"},
+            {"type": "text", "text": "Transcribe and summarise this audio."},
+        ],
+    },
+]
+response = pipe.chat(messages)
+```
+
+## Deployment (vLLM-Omni)
+
+####
+
+# 1. Clone & Build
+```bash
+git clone https://github.com/krafton-ai/vllm-omni.git
+cd vllm-omni
+docker build -f docker/Dockerfile.ci -t vllm-omni .
+```
+
+# 2. Serve
+```bash
+docker run --rm --gpus all \
+  --shm-size=16g \
+  -p 8000:8000 \
+  vllm-omni \
+  bash -c "vllm serve KRAFTON/Raon-Speech-9B --omni --port 8000 --trust-remote-code"
+```
+
+# 3. Test — TTS
+```bash
+curl -X POST http://localhost:8000/v1/audio/speech \
+  -H "Content-Type: application/json" \
+  -d '{
+    "input": "Hello, how are you?",
+    "model": "KRAFTON/Raon-Speech-9B",
+    "response_format": "wav"
+  }' --output output.wav
+```
+
+# 4. Test — TTS with voice cloning
+```bash
+curl -X POST http://localhost:8000/v1/audio/speech \
+  -H "Content-Type: application/json" \
+  -d '{
+    "input": "Hello, how are you?",
+    "model": "KRAFTON/Raon-Speech-9B",
+    "ref_audio": "data:audio/wav;base64,'$(base64 -w0 speaker_ref.wav)'",
+    "task_type": "Base",
+    "response_format": "wav"
+  }' --output cloned.wav
+```
+
+
+## License
+
+This repository is licensed under the
+[Creative Commons Attribution-NonCommercial 4.0 International License](https://creativecommons.org/licenses/by-nc/4.0/).
+
+© 2026 KRAFTON

added_tokens.json

@@ -0,0 +1,21 @@
+{
+  "<tts_pad>": 151691,
+  "<tts_text_bos>": 151680,
+  "<tts_text_bos_single>": 151682,
+  "<tts_text_eod>": 151681,
+  "<|audio_end|>": 151670,
+  "<|audio_input_placeholder|>": 151676,
+  "<|audio_output_end_pad|>": 151678,
+  "<|audio_output_pad|>": 151677,
+  "<|audio_output_placeholder|>": 151675,
+  "<|audio_pad|>": 151683,
+  "<|box_end|>": 151688,
+  "<|box_start|>": 151687,
+  "<|endoftext|>": 151679,
+  "<|object_ref_end|>": 151686,
+  "<|object_ref_start|>": 151685,
+  "<|quad_end|>": 151690,
+  "<|quad_start|>": 151689,
+  "<|secondary_audio_pad|>": 151684,
+  "<|speaker_embedding_placeholder|>": 151671
+}

chat_template.jinja

@@ -0,0 +1,121 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {{- "<|vision_start|><|image_pad|><|vision_end|>" }}
+                {%- elif content.type == 'audio' or 'audio' in content or 'audio_url' in content %}
+                    {{- "<|audio_start|><|audio_pad|><|audio_end|>" }}
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {{- "<|vision_start|><|video_pad|><|vision_end|>" }}
+                {%- elif content.type == 'text' %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+    {{- '\n\n' }}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {{- '<|im_start|>system\n' +"<|vision_start|><|image_pad|><|vision_end|>"+ '<|im_end|>\n' }}
+                {%- elif content.type == 'audio' or 'audio' in content or 'audio_url' in content %}
+                    {{- '<|im_start|>system\n' +"<|audio_start|><|audio_pad|><|audio_end|>"+ '<|im_end|>\n' }}
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {{- '<|im_start|>system\n' +"<|vision_start|><|video_pad|><|vision_end|>"+ '<|im_end|>\n' }}
+                {%- elif content.type == 'text' %}
+                    {{- '<|im_start|>system\n' +content.text+ '<|im_end|>\n' }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = namespace(text="") %}
+        {%- for mcontent in message.content %}
+            {%- if mcontent.type == 'image' or 'image' in mcontent or 'image_url' in mcontent %}
+                {%- set content.text = content.text~"<|vision_start|><|image_pad|><|vision_end|>" %}
+            {%- elif mcontent.type == 'audio' or 'audio' in mcontent or 'audio_url' in mcontent %}
+                {%- set content.text = content.text~"<|audio_start|><|audio_pad|><|audio_end|>" %}
+            {%- elif mcontent.type == 'video' or 'video' in mcontent %}
+                {%- set content.text = content.text~"<|vision_start|><|video_pad|><|vision_end|>" %}
+            {%- elif mcontent.type == 'text' %}
+                {%- set content.text = content.text~mcontent.text %}
+            {%- endif %}
+        {%- endfor %}
+        {%- set content = content.text %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = "" %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+            {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+            {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+        {%- endif %}
+    {%- endif %}
+    {%- if loop.index0 > ns.last_query_index %}
+        {%- if loop.last or (not loop.last and reasoning_content) %}
+            {{- '<|im_start|>' + message.role + '\n' + reasoning_content + content.lstrip('\n') }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+    {%- else %}
+        {{- '<|im_start|>' + message.role + '\n' + content }}
+    {%- endif %}
+    {%- if message.tool_calls %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if (loop.first and content) or (not loop.first) %}{{- '\n' }}{%- endif %}
+            {%- if tool_call.function %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {%- if tool_call.arguments is string %}
+                {{- tool_call.arguments }}
+            {%- else %}
+                {{- tool_call.arguments | tojson }}
+            {%- endif %}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}{{- '<|im_start|>user' }}{%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}{{- '<|im_end|>\n' }}{%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config.json

@@ -0,0 +1,795 @@
+{
+  "_name_or_path": "",
+  "accept_hidden_layer": -1,
+  "acoustic_delay": null,
+  "add_cross_attention": false,
+  "architectures": [
+    "RaonModel"
+  ],
+  "audio_encoder_config": {
+    "_name_or_path": "",
+    "activation_dropout": 0,
+    "activation_function": "gelu",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "conv_chunksize": 500,
+    "cross_attention_hidden_size": null,
+    "d_model": 1024,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "downsample_hidden_size": 480,
+    "dropout": 0,
+    "dtype": "bfloat16",
+    "early_stopping": false,
+    "encoder_attention_heads": 16,
+    "encoder_ffn_dim": 4096,
+    "encoder_layers": 24,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_source_positions": 1500,
+    "min_length": 0,
+    "model_type": "qwen3_omni_moe_audio_encoder",
+    "n_window": 50,
+    "n_window_infer": 800,
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 24,
+    "num_mel_bins": 128,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_dim": 2048,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 151679,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sampling_rate": 24000,
+    "scale_embedding": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false
+  },
+  "audio_tokenizer_config": {
+    "_frame_rate": 12.5,
+    "_name_or_path": "kyutai/mimi",
+    "add_cross_attention": false,
+    "architectures": [
+      "MimiModel"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "audio_channels": 1,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "codebook_dim": 256,
+    "codebook_size": 2048,
+    "compress": 2,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "dilation_growth_rate": 2,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": "bfloat16",
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_dim": 64,
+    "hidden_act": "gelu",
+    "hidden_size": 512,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 2048,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "kernel_size": 7,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "last_kernel_size": 3,
+    "layer_scale_initial_scale": 0.01,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 8000,
+    "min_length": 0,
+    "model_type": "mimi",
+    "no_repeat_ngram_size": 0,
+    "norm_eps": 1e-05,
+    "normalize": false,
+    "num_attention_heads": 8,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_filters": 64,
+    "num_hidden_layers": 8,
+    "num_key_value_heads": 8,
+    "num_quantizers": 32,
+    "num_residual_layers": 1,
+    "num_return_sequences": 1,
+    "num_semantic_quantizers": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_mode": "constant",
+    "pad_token_id": 151679,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "residual_kernel_size": 3,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rope_theta": 10000.0,
+    "sampling_rate": 24000,
+    "sep_token_id": null,
+    "sliding_window": 250,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "trim_right_ratio": 1.0,
+    "typical_p": 1.0,
+    "upsample_groups": 512,
+    "upsampling_ratios": [
+      8,
+      6,
+      5,
+      4
+    ],
+    "use_bfloat16": true,
+    "use_cache": false,
+    "use_causal_conv": true,
+    "use_conv_shortcut": false,
+    "use_streaming": false,
+    "vector_quantization_hidden_dimension": 256
+  },
+  "aut_is_causal": false,
+  "bad_words_ids": null,
+  "begin_suppress_tokens": null,
+  "bos_token_id": null,
+  "chunk_size_feed_forward": 0,
+  "code_predictor_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_bias": false,
+    "attention_dropout": 0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": "bfloat16",
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 32768,
+    "max_window_layers": 28,
+    "min_length": 0,
+    "model_type": "qwen3_omni_moe_talker_code_predictor",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_code_groups": 16,
+    "num_hidden_layers": 5,
+    "num_key_value_heads": 8,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 151679,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 1000000,
+    "sep_token_id": null,
+    "sliding_window": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": true,
+    "use_cache": false,
+    "use_sliding_window": false,
+    "vocab_size": 2048
+  },
+  "cross_attention_hidden_size": null,
+  "decoder_start_token_id": null,
+  "depth_loss_weight": 0.25,
+  "diversity_penalty": 0.0,
+  "do_sample": false,
+  "dtype": "bfloat16",
+  "early_stopping": false,
+  "encoder_no_repeat_ngram_size": 0,
+  "eos_token_id": null,
+  "exponential_decay_length_penalty": null,
+  "finetuning_task": null,
+  "forced_bos_token_id": null,
+  "forced_eos_token_id": null,
+  "hidden_size": null,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1"
+  },
+  "input_adaptor_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_config": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": "bfloat16",
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_size": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "input_size": 2048,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "embedding_adaptor",
+    "no_repeat_ngram_size": 0,
+    "norm_eps": 1e-06,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_layers": 2,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "output_size": 4096,
+    "output_time_scale": 1,
+    "pad_token_id": null,
+    "post_norm_init_scale": 0.02,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_post_norm": true
+  },
+  "input_num_code_groups": null,
+  "is_decoder": false,
+  "is_encoder_decoder": false,
+  "keys_to_ignore_at_inference": [
+    "past_key_values"
+  ],
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1
+  },
+  "length_penalty": 1.0,
+  "max_length": 20,
+  "min_length": 0,
+  "model_type": "raon",
+  "no_repeat_ngram_size": 0,
+  "num_beam_groups": 1,
+  "num_beams": 1,
+  "num_return_sequences": 1,
+  "num_talker_layers": 4,
+  "output_adaptor_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_config": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": "bfloat16",
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_size": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "input_size": 512,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "embedding_adaptor",
+    "no_repeat_ngram_size": 0,
+    "norm_eps": 1e-06,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_layers": 2,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "output_size": 4096,
+    "output_time_scale": 1,
+    "pad_token_id": null,
+    "post_norm_init_scale": 0.02,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_post_norm": true
+  },
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "output_scores": false,
+  "pad_token_id": 151679,
+  "prefix": null,
+  "problem_type": null,
+  "proj_code_bias": true,
+  "pruned_heads": {},
+  "remove_invalid_values": false,
+  "repetition_penalty": 1.0,
+  "return_dict": true,
+  "return_dict_in_generate": false,
+  "sep_token_id": null,
+  "speaker_encoder_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": null,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "encoder_type": "ecapa_tdnn",
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "frame_rate": 12.5,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "input_size": 512,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_seconds": 10.0,
+    "min_length": 0,
+    "min_seconds": 2.0,
+    "model_type": "speaker_encoder",
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_heads": 8,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "output_size": 4096,
+    "pad_token_id": null,
+    "prefix": null,
+    "pretrained_dim": 192,
+    "pretrained_model_id": "speechbrain/spkrec-ecapa-voxceleb",
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false
+  },
+  "supports_audio_input": true,
+  "supports_audio_output": true,
+  "suppress_tokens": null,
+  "talker_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": null,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 6144,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 32768,
+    "max_window_layers": 28,
+    "min_length": 0,
+    "model_type": "qwen3",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 4,
+    "num_key_value_heads": 8,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 151679,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 10000.0,
+    "sep_token_id": null,
+    "sliding_window": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 153723
+  },
+  "task_specific_params": null,
+  "temperature": 1.0,
+  "text_model_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": [
+      "Qwen3ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dtype": "bfloat16",
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 12288,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 262144,
+    "max_window_layers": 36,
+    "min_length": 0,
+    "model_type": "qwen3",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 32,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 36,
+    "num_key_value_heads": 8,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 5000000,
+    "sep_token_id": null,
+    "sliding_window": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": true,
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 153723
+  },
+  "tf_legacy_loss": false,
+  "thinker_to_talker_intermediate_size": 6144,
+  "thinker_to_talker_pre_norm": false,
+  "thinker_to_talker_projection_mode": "mlp",
+  "tie_encoder_decoder": false,
+  "tie_word_embeddings": true,
+  "tokenizer_class": null,
+  "top_k": 50,
+  "top_p": 1.0,
+  "torchscript": false,
+  "transformers_version": "4.57.3",
+  "typical_p": 1.0,
+  "use_bfloat16": false,
+  "use_duplex_end_pad": false,
+  "use_inline_text_prediction": false,
+  "auto_map": {
+    "AutoConfig": "configuration_raon.RaonConfig",
+    "AutoModel": "modeling_raon.RaonModel"
+  }
+}

configuration_raon.py

@@ -0,0 +1,510 @@
+# AUTO-GENERATED — do not edit manually. Run build_hub_files.py to regenerate.
+from __future__ import annotations
+
+from copy import deepcopy
+from typing import Any
+
+from transformers import PretrainedConfig, MimiConfig, Qwen3Config
+from transformers.models.qwen3_omni_moe.configuration_qwen3_omni_moe import (
+    Qwen3OmniMoeAudioEncoderConfig,
+    Qwen3OmniMoeTalkerCodePredictorConfig,
+    Qwen3OmniMoeTextConfig,
+)
+
+# ── from modules/embedding.py ──
+
+class EmbeddingAdaptorConfig(PretrainedConfig):
+    """Configuration for EmbeddingAdaptor.
+
+    Controls the projection from audio encoder embeddings to LM embedding space,
+    including the time-scale ratio, MLP depth, optional transformer decoder, and
+    optional post-projection RMSNorm.
+
+    Args:
+        input_size: Feature dimension of the encoder output (e.g. 512 for Mimi).
+        output_size: Feature dimension expected by the LM (e.g. 4096 for Qwen3-7B).
+        output_time_scale: Ratio of output frames to input frames. Values >= 1
+            upsample (expand time); values < 1 downsample (compress time).
+            Must be a reciprocal integer in either direction.
+        num_layers: Number of MLP layers (1 or 2). Ignored in transformer mode.
+        hidden_size: Hidden dimension for the 2-layer MLP. Defaults to output_size.
+        decoder_config: If provided, uses a lightweight Qwen3 transformer instead
+            of an MLP for the adaptor projection.
+        use_post_norm: If True, apply RMSNorm to the output embeddings.
+        norm_eps: Epsilon for RMSNorm.
+        post_norm_init_scale: If set, initialize RMSNorm weight to this value
+            (useful for residual scaling at initialisation).
+    """
+
+    model_type = "embedding_adaptor"
+
+    def __init__(
+        self,
+        input_size: int = 512,
+        output_size: int = 4096,
+        output_time_scale: float = 1.0,
+        num_layers: int = 1,
+        hidden_size: int | None = None,
+        decoder_config: dict[str, Any] | Qwen3Config | None = None,
+        use_post_norm: bool = False,
+        norm_eps: float = 1e-6,
+        post_norm_init_scale: float | None = None,
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.input_size = input_size
+        self.output_size = output_size
+        self.output_time_scale = output_time_scale
+        self.num_layers = num_layers
+        self.hidden_size = hidden_size
+        self.use_post_norm = use_post_norm
+        self.norm_eps = norm_eps
+        self.post_norm_init_scale = post_norm_init_scale
+
+        # Parse decoder_config for transformer adaptor mode
+        if isinstance(decoder_config, dict):
+            decoder_config = Qwen3Config(**decoder_config)
+        self.decoder_config = decoder_config
+
+
+# ── from modules/speaker_encoder.py ──
+
+class SpeakerEncoderConfig(PretrainedConfig):
+    """Configuration for SpeakerEncoder: input/output sizes, attention heads, and frame window."""
+
+    model_type = "speaker_encoder"
+
+    def __init__(
+        self,
+        input_size: int = 512,
+        output_size: int = 4096,
+        num_heads: int = 8,
+        min_seconds: float = 2.0,
+        max_seconds: float = 10.0,
+        frame_rate: float = 12.5,
+        encoder_type: str = "from_scratch",
+        pretrained_model_id: str | None = None,
+        pretrained_dim: int | None = None,
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.input_size = input_size
+        self.output_size = output_size
+        self.num_heads = num_heads
+        self.min_seconds = min_seconds
+        self.max_seconds = max_seconds
+        self.frame_rate = frame_rate
+        self.encoder_type = encoder_type
+        self.pretrained_model_id = pretrained_model_id
+        self.pretrained_dim = pretrained_dim
+
+
+# ── from modules/voxtral_encoder.py ──
+
+class VoxtralRealtimeEncoderConfig(PretrainedConfig):
+    """Configuration for the Voxtral Realtime audio encoder.
+
+    Stores both the encoder architecture parameters and the projector/downsample
+    settings needed to reconstruct the full audio pipeline.
+    """
+
+    model_type = "voxtral_realtime_encoder"
+
+    def __init__(
+        self,
+        hidden_size: int = 1280,
+        intermediate_size: int = 5120,
+        num_hidden_layers: int = 32,
+        num_attention_heads: int = 32,
+        num_key_value_heads: int | None = None,
+        activation_function: str = "gelu",
+        num_mel_bins: int = 128,
+        initializer_range: float = 0.02,
+        attention_dropout: float = 0.0,
+        hidden_act: str = "silu",
+        max_position_embeddings: int = 1500,
+        rms_norm_eps: float = 1e-5,
+        rope_theta: float = 10000.0,
+        sliding_window: int = 750,
+        head_dim: int = 64,
+        downsample_factor: int = 4,
+        projector_hidden_act: str = "gelu",
+        projector_output_size: int | None = None,
+        output_embedding_scale: float = 1.0,
+        skip_projector: bool = False,
+        attn_implementation: str = "eager",
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads if num_key_value_heads is not None else num_attention_heads
+        self.activation_function = activation_function
+        self.num_mel_bins = num_mel_bins
+        self.initializer_range = initializer_range
+        self.attention_dropout = attention_dropout
+        self.hidden_act = hidden_act
+        self.max_position_embeddings = max_position_embeddings
+        self.rms_norm_eps = rms_norm_eps
+        self.rope_theta = rope_theta
+        self.sliding_window = sliding_window
+        self.head_dim = head_dim if head_dim is not None else hidden_size // num_attention_heads
+        self.downsample_factor = downsample_factor
+        self.projector_hidden_act = projector_hidden_act
+        self.projector_output_size = projector_output_size
+        self.output_embedding_scale = output_embedding_scale
+        self.skip_projector = skip_projector
+        self._attn_implementation = attn_implementation
+
+        # Aliases expected by the encoder layers.
+        self.encoder_layers = num_hidden_layers
+        self.encoder_attention_heads = num_attention_heads
+
+    @classmethod
+    def from_pretrained(
+        cls,
+        pretrained_model_name_or_path: str,
+        **kwargs: Any,
+    ) -> "VoxtralRealtimeEncoderConfig":
+        """Load config from a Voxtral Realtime checkpoint.
+
+        Reads ``config.json`` and extracts the ``audio_config`` sub-dict along
+        with top-level ``downsample_factor``, ``projector_hidden_act``, and
+        ``text_config.hidden_size`` (used as ``projector_output_size``).
+
+        Works with both the full ``voxtral_realtime`` model config and a
+        standalone ``voxtral_realtime_encoder`` config.
+
+        Args:
+            pretrained_model_name_or_path: HuggingFace model ID or local path.
+
+        Returns:
+            Populated ``VoxtralRealtimeEncoderConfig``.
+        """
+        import json
+        import os
+
+        from huggingface_hub import hf_hub_download
+
+        is_local = os.path.isdir(pretrained_model_name_or_path)
+        if is_local:
+            config_path = os.path.join(pretrained_model_name_or_path, "config.json")
+        else:
+            config_path = hf_hub_download(
+                repo_id=pretrained_model_name_or_path,
+                filename="config.json",
+            )
+
+        with open(config_path) as f:
+            full_config = json.load(f)
+
+        # If this is the full model config, extract the audio sub-config.
+        if "audio_config" in full_config:
+            audio_cfg = full_config["audio_config"]
+            downsample_factor = full_config.get("downsample_factor", 4)
+            projector_hidden_act = full_config.get("projector_hidden_act", "gelu")
+            text_hidden_size = full_config.get("text_config", {}).get("hidden_size")
+        else:
+            # Standalone encoder config (e.g. saved by us).
+            audio_cfg = full_config
+            downsample_factor = audio_cfg.get("downsample_factor", 4)
+            projector_hidden_act = audio_cfg.get("projector_hidden_act", "gelu")
+            text_hidden_size = audio_cfg.get("projector_output_size")
+
+        # The upstream rope_theta lives inside rope_parameters.
+        rope_params = audio_cfg.get("rope_parameters") or {}
+        rope_theta = rope_params.get("rope_theta", audio_cfg.get("rope_theta", 10000.0))
+
+        return cls(
+            hidden_size=audio_cfg.get("hidden_size", 1280),
+            intermediate_size=audio_cfg.get("intermediate_size", 5120),
+            num_hidden_layers=audio_cfg.get("num_hidden_layers", 32),
+            num_attention_heads=audio_cfg.get("num_attention_heads", 32),
+            num_key_value_heads=audio_cfg.get("num_key_value_heads"),
+            activation_function=audio_cfg.get("activation_function", "gelu"),
+            num_mel_bins=audio_cfg.get("num_mel_bins", 128),
+            initializer_range=audio_cfg.get("initializer_range", 0.02),
+            attention_dropout=audio_cfg.get("attention_dropout", 0.0),
+            hidden_act=audio_cfg.get("hidden_act", "silu"),
+            max_position_embeddings=audio_cfg.get("max_position_embeddings", 1500),
+            rms_norm_eps=audio_cfg.get("rms_norm_eps", 1e-5),
+            rope_theta=rope_theta,
+            sliding_window=audio_cfg.get("sliding_window", 750),
+            head_dim=audio_cfg.get("head_dim", 64),
+            downsample_factor=downsample_factor,
+            projector_hidden_act=projector_hidden_act,
+            projector_output_size=text_hidden_size,
+            **kwargs,
+        )
+
+
+# ---------------------------------------------------------------------------
+# Conv1d padding cache (for streaming)
+# ---------------------------------------------------------------------------
+
+
+# ── from models/raon.py ──
+
+TEXT_MODEL_CONFIGS: dict[str, type[PretrainedConfig]] = {
+    Qwen3Config.model_type: Qwen3Config,
+}
+
+
+class RaonConfig(PretrainedConfig):
+    """Configuration class for RaonModel."""
+
+    model_type = "raon"
+    has_no_defaults_at_init = True
+    text_model_config: PretrainedConfig = None
+    audio_encoder_config: Qwen3OmniMoeAudioEncoderConfig | VoxtralRealtimeEncoderConfig = None
+    audio_tokenizer_config: MimiConfig = None
+    input_adaptor_config: EmbeddingAdaptorConfig = None
+    output_adaptor_config: EmbeddingAdaptorConfig = None
+    code_predictor_config: Qwen3OmniMoeTalkerCodePredictorConfig = None
+    speaker_encoder_config: SpeakerEncoderConfig | None = None
+    # Note: speaker_encoder_config is intentionally excluded from sub_configs.
+    # It is optional (can be None), and transformers' _get_dtype unconditionally
+    # calls sub_config.dtype on every entry, which crashes on None.
+    # Deserialization from dict is handled in __init__ instead.
+    sub_configs = {
+        "text_model_config": PretrainedConfig,
+        "audio_encoder_config": PretrainedConfig,
+        "audio_tokenizer_config": PretrainedConfig,
+        "input_adaptor_config": EmbeddingAdaptorConfig,
+        "output_adaptor_config": EmbeddingAdaptorConfig,
+        "code_predictor_config": Qwen3OmniMoeTalkerCodePredictorConfig,
+    }
+
+    def __init__(
+        self,
+        *,
+        text_model_config: dict[str, Any] | PretrainedConfig | None = None,
+        audio_encoder_config: dict[str, Any] | Qwen3OmniMoeAudioEncoderConfig | VoxtralRealtimeEncoderConfig | None = None,
+        audio_tokenizer_config: dict[str, Any] | MimiConfig | None = None,
+        input_adaptor_config: dict[str, Any] | EmbeddingAdaptorConfig | None = None,
+        output_adaptor_config: dict[str, Any] | EmbeddingAdaptorConfig | None = None,
+        code_predictor_config: dict[str, Any] | Qwen3OmniMoeTalkerCodePredictorConfig | None = None,
+        speaker_encoder_config: dict[str, Any] | SpeakerEncoderConfig | None = None,
+        num_talker_layers: int = 0,
+        supports_audio_input: bool = True,
+        supports_audio_output: bool = True,
+        aut_is_causal: bool = False,
+        proj_code_bias: bool = False,
+        accept_hidden_layer: int = -1,
+        talker_config: dict[str, Any] | PretrainedConfig | None = None,
+        thinker_to_talker_pre_norm: bool = False,
+        sequence_mode: str | None = None,
+        use_sil_token: bool = False,
+        no_audio_in_sil: bool = False,
+        text_lookahead: int = 0,
+        use_duplex_end_pad: bool = False,
+        speaker_embedding_to_code_predictor: bool = True,
+        duplex_pad_token_id: int | None = None,
+        duplex_end_pad_token_id: int | None = None,
+        duplex_sil_token_id: int | None = None,
+        duplex_bc_token_id: int | None = None,
+        use_backchannel_token: bool = False,
+        bc_loss_weight: float = 1.0,
+        speaker_token_id: int | None = None,
+        audio_input_token_id: int | None = None,
+        audio_output_token_id: int | None = None,
+        audio_start_token_id: int | None = None,
+        im_start_token_id: int | None = None,
+        text_loss_weight: float = 1.0,
+        sil_loss_weight: float = 1.0,
+        epad_loss_weight: float = 0.0,
+        semantic_loss_weight: float = 1.0,
+        acoustic_loss_weights: list[float] | None = None,
+        audio_lm_head_enabled: bool = True,
+        delays: list[int] | None = None,
+        **kwargs: Any,
+    ) -> None:
+        super().__init__(**kwargs)
+
+        # Ensure auto_map is always serialized for trust_remote_code Hub loading.
+        if not hasattr(self, "auto_map") or not self.auto_map:
+            self.auto_map = {
+                "AutoConfig": "configuration_raon.RaonConfig",
+                "AutoModel": "modeling_raon.RaonModel",
+            }
+
+        assert text_model_config is not None, "RaonConfig: `text_model_config` is required."
+        assert audio_encoder_config is not None, "RaonConfig: `audio_encoder_config` is required."
+        assert audio_tokenizer_config is not None, "RaonConfig: `audio_tokenizer_config` is required."
+        assert input_adaptor_config is not None, "RaonConfig: `input_adaptor_config` is required."
+        assert output_adaptor_config is not None, "RaonConfig: `output_adaptor_config` is required."
+        assert code_predictor_config is not None, "RaonConfig: `code_predictor_config` is required."
+
+        if isinstance(text_model_config, dict):
+            model_type = text_model_config.get("model_type", Qwen3Config.model_type)
+            text_model_config = TEXT_MODEL_CONFIGS[model_type](**text_model_config)
+
+        # Convert sub-configs from dict or generic PretrainedConfig to specific types.
+        # The generic PretrainedConfig case occurs when transformers' sub_configs mechanism
+        # auto-deserializes before __init__ runs (e.g. with trust_remote_code Hub loading).
+        def _to_dict(cfg: Any) -> dict[str, Any]:
+            """Convert a config to dict, handling both dict and PretrainedConfig."""
+            if isinstance(cfg, dict):
+                return cfg
+            return cfg.to_dict()
+
+        if isinstance(audio_encoder_config, dict) or (
+            isinstance(audio_encoder_config, PretrainedConfig)
+            and not isinstance(audio_encoder_config, (Qwen3OmniMoeAudioEncoderConfig, VoxtralRealtimeEncoderConfig))
+        ):
+            d = _to_dict(audio_encoder_config)
+            model_type = d.get("model_type", Qwen3OmniMoeAudioEncoderConfig.model_type)
+            if model_type == Qwen3OmniMoeAudioEncoderConfig.model_type:
+                audio_encoder_config = Qwen3OmniMoeAudioEncoderConfig(**d)
+            elif model_type == "voxtral_realtime_encoder":
+                audio_encoder_config = VoxtralRealtimeEncoderConfig(**d)
+            else:
+                raise ValueError(
+                    f"Unsupported audio_encoder model_type: {model_type!r}. "
+                    "Expected 'qwen3_omni_moe_audio_encoder' or 'voxtral_realtime_encoder'."
+                )
+
+        if isinstance(audio_tokenizer_config, dict) or (
+            isinstance(audio_tokenizer_config, PretrainedConfig) and not isinstance(audio_tokenizer_config, MimiConfig)
+        ):
+            audio_tokenizer_config = MimiConfig(**_to_dict(audio_tokenizer_config))
+
+        if isinstance(input_adaptor_config, dict) or (
+            isinstance(input_adaptor_config, PretrainedConfig)
+            and not isinstance(input_adaptor_config, EmbeddingAdaptorConfig)
+        ):
+            input_adaptor_config = EmbeddingAdaptorConfig(**_to_dict(input_adaptor_config))
+
+        if isinstance(output_adaptor_config, dict) or (
+            isinstance(output_adaptor_config, PretrainedConfig)
+            and not isinstance(output_adaptor_config, EmbeddingAdaptorConfig)
+        ):
+            output_adaptor_config = EmbeddingAdaptorConfig(**_to_dict(output_adaptor_config))
+
+        if isinstance(code_predictor_config, dict) or (
+            isinstance(code_predictor_config, PretrainedConfig)
+            and not isinstance(code_predictor_config, Qwen3OmniMoeTalkerCodePredictorConfig)
+        ):
+            code_predictor_config = Qwen3OmniMoeTalkerCodePredictorConfig(**_to_dict(code_predictor_config))
+
+        if isinstance(speaker_encoder_config, dict) or (
+            isinstance(speaker_encoder_config, PretrainedConfig)
+            and not isinstance(speaker_encoder_config, SpeakerEncoderConfig)
+        ):
+            speaker_encoder_config = SpeakerEncoderConfig(**_to_dict(speaker_encoder_config))
+
+        if isinstance(talker_config, dict) or (
+            isinstance(talker_config, PretrainedConfig) and type(talker_config) is PretrainedConfig
+        ):
+            d = _to_dict(talker_config) if talker_config is not None else {}
+            talker_model_type = d.get("model_type", Qwen3Config.model_type)
+            talker_config = TEXT_MODEL_CONFIGS[talker_model_type](**d)
+
+        assert isinstance(
+            audio_encoder_config, (Qwen3OmniMoeAudioEncoderConfig, VoxtralRealtimeEncoderConfig, MimiConfig)
+        ), "audio_encoder_config must be Qwen3OmniMoeAudioEncoderConfig, VoxtralRealtimeEncoderConfig, or MimiConfig."
+        assert isinstance(audio_tokenizer_config, MimiConfig), "audio_tokenizer_config must be MimiConfig."
+        assert isinstance(input_adaptor_config, EmbeddingAdaptorConfig), (
+            "input_adaptor_config must be EmbeddingAdaptorConfig."
+        )
+        assert isinstance(output_adaptor_config, EmbeddingAdaptorConfig), (
+            "output_adaptor_config must be EmbeddingAdaptorConfig."
+        )
+        assert isinstance(code_predictor_config, Qwen3OmniMoeTalkerCodePredictorConfig), (
+            "code_predictor_config must be Qwen3OmniMoeTalkerCodePredictorConfig."
+        )
+        assert isinstance(text_model_config, PretrainedConfig), "text_model_config must be PretrainedConfig."
+        assert speaker_encoder_config is None or isinstance(speaker_encoder_config, SpeakerEncoderConfig), (
+            "speaker_encoder_config must be None or SpeakerEncoderConfig."
+        )
+
+        self.text_model_config = text_model_config
+        self.audio_encoder_config = audio_encoder_config
+        self.audio_tokenizer_config = audio_tokenizer_config
+        self.input_adaptor_config = input_adaptor_config
+        self.output_adaptor_config = output_adaptor_config
+        self.code_predictor_config = code_predictor_config
+        self.speaker_encoder_config = speaker_encoder_config
+        self.num_talker_layers = num_talker_layers
+        self.supports_audio_input = supports_audio_input
+        self.supports_audio_output = supports_audio_output
+        self.aut_is_causal = aut_is_causal
+        self.proj_code_bias = proj_code_bias
+        self.accept_hidden_layer = accept_hidden_layer
+        self.talker_config = talker_config
+        self.thinker_to_talker_pre_norm = thinker_to_talker_pre_norm
+        self.sequence_mode = sequence_mode
+        self.use_sil_token = use_sil_token
+        self.no_audio_in_sil = no_audio_in_sil
+        self.text_lookahead = int(text_lookahead)
+        self.use_duplex_end_pad = use_duplex_end_pad
+        self.speaker_embedding_to_code_predictor = speaker_embedding_to_code_predictor
+        self.duplex_pad_token_id = duplex_pad_token_id
+        self.duplex_end_pad_token_id = duplex_end_pad_token_id
+        self.duplex_sil_token_id = duplex_sil_token_id
+        self.duplex_bc_token_id = duplex_bc_token_id
+        self.use_backchannel_token = use_backchannel_token
+        self.bc_loss_weight = bc_loss_weight
+        self.speaker_token_id = speaker_token_id
+        self.audio_input_token_id = audio_input_token_id
+        self.audio_output_token_id = audio_output_token_id
+        self.audio_start_token_id = audio_start_token_id
+        self.im_start_token_id = im_start_token_id
+        self.text_loss_weight = text_loss_weight
+        self.sil_loss_weight = sil_loss_weight
+        self.epad_loss_weight = epad_loss_weight
+        self.semantic_loss_weight = semantic_loss_weight
+        self.acoustic_loss_weights = acoustic_loss_weights
+        self.audio_lm_head_enabled = audio_lm_head_enabled
+        self.delays = delays
+
+        if supports_audio_output and audio_lm_head_enabled:
+            assert talker_config is not None, "RaonConfig: `talker_config` is required when audio output is enabled."
+            assert num_talker_layers > 0, "RaonConfig: `num_talker_layers` must be positive when audio output is enabled."
+
+    def _get_non_default_generation_parameters(self) -> dict[str, Any]:
+        return {}
+
+    def to_diff_dict(self) -> dict[str, Any]:
+        """Return config as a dict suitable for diffing."""
+        return self.to_dict()
+
+
+class RaonDuplexConfig(RaonConfig):
+    """Configuration alias for full-duplex checkpoints (model_type='raon_duplex')."""
+
+    model_type = "raon_duplex"
+
+    def __init__(self, **kwargs: Any) -> None:
+        # Duplex-specific defaults; overridden by values in config.json when present.
+        kwargs.setdefault("sequence_mode", "uta")
+        kwargs.setdefault("use_sil_token", True)
+        kwargs.setdefault("no_audio_in_sil", False)
+        kwargs.setdefault("text_lookahead", 0)
+        kwargs.setdefault("use_duplex_end_pad", True)
+        kwargs.setdefault("duplex_pad_token_id", 151677)
+        kwargs.setdefault("duplex_end_pad_token_id", 151678)
+        kwargs.setdefault("duplex_sil_token_id", 151672)
+        kwargs.setdefault("duplex_bc_token_id", 151673)
+        kwargs.setdefault("speaker_token_id", 151671)
+        kwargs.setdefault("audio_input_token_id", 151676)
+        kwargs.setdefault("audio_output_token_id", 151675)
+        kwargs.setdefault("audio_start_token_id", 151669)
+        kwargs.setdefault("im_start_token_id", 151644)
+        # Loss weight defaults (overridable at training time via duplex_train args)
+        kwargs.setdefault("text_loss_weight", 1.0)
+        kwargs.setdefault("sil_loss_weight", 1.0)
+        kwargs.setdefault("epad_loss_weight", 0.0)
+        kwargs.setdefault("semantic_loss_weight", 1.0)
+        kwargs.setdefault("acoustic_loss_weights", None)
+        super().__init__(**kwargs)
+        self.auto_map = {
+            "AutoConfig": "configuration_raon.RaonDuplexConfig",
+            "AutoModel": "modeling_raon.RaonDuplexModel",
+        }
+
+
+# Duplex model — same architecture, different model_type for HF registry
+

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dff45b01197c041324d528c65d5551670c1292aa40874beea4c571bfaae2d7c5
+size 4916897336

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e24f7df5232cf22e66b276bf3f545c78d649fd53fd66e9282a65da5fda3277d
+size 4915961072

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20c5e7c016455faa9a854af8e6b2f9b391e5a59facd603b70ea427db522a3bb2
+size 4983069200

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7776e8b1c4bb0a1ea47c1adb98d689cd488164bc067dfea1e8bfc5aeb4a964ab
+size 3290040026

special_tokens_map.json

@@ -0,0 +1,26 @@
+{
+  "additional_special_tokens": [
+    "<|audio_input_placeholder|>"
+  ],
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "image_token": "<|image_pad|>",
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c865b4da59eb914ef2e574352533b8b09c8f8ae7c5ef64b0a3ddb35d884e3d00
+size 11426101

tokenizer_config.json

@@ -0,0 +1,346 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|audio_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "<|audio_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "<|speaker_embedding_placeholder|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151675": {
+      "content": "<|audio_output_placeholder|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151676": {
+      "content": "<|audio_input_placeholder|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151677": {
+      "content": "<|audio_output_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151678": {
+      "content": "<|audio_output_end_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151679": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151680": {
+      "content": "<tts_text_bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151681": {
+      "content": "<tts_text_eod>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151682": {
+      "content": "<tts_text_bos_single>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151683": {
+      "content": "<|audio_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151684": {
+      "content": "<|secondary_audio_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151685": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151686": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151687": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151688": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151689": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151690": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151691": {
+      "content": "<tts_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|audio_input_placeholder|>"
+  ],
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "image_token": "<|image_pad|>",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}