README.md

---

language:
  - ru
  - zh 
  - en 
  - de
  - es
  - fr
  - ja
  - it
  - pt
  - ko
tags:
  - text-to-speech
  - TTS
  - ONNX
  - qwen3-tts
  - voice-clone
  - streaming
  - qwen3
  - vq
  - rvq
  - ecapa-tdnn
  - multilingual
pipeline_tag: text-to-speech
license: apache-2.0
base_model: Qwen/Qwen3-TTS-12Hz-0.6B-Base
---


# Qwen3-TTS-Streaming ONNX Inference

Pure ONNX Runtime inference pipeline for [Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base), enabling **real-time streaming text-to-speech** without PyTorch dependency at runtime.

## Overview

This repository provides:

- **`qwen3_tts_inferencer_onnx.py`** — Core streaming TTS engine that orchestrates six ONNX models (talker LLM, local talker transformer, codec decoder, speaker encoder, talker codec embedding, text embedding projection) using only NumPy and ONNX Runtime.

- **`test_qwen3-tts-streaming_onnx.py`** — End-to-end test script that simulates LLM streaming text and produces a WAV file.



## Architecture



```

Reference Audio ──► Speaker Encoder ──► Speaker Embedding Vector (voice clone context)

                                           │

                                           ▼

            Text Deltas ──► Talker LLM (Qwen3-0.6B) ──► [Hidden States, VQ Token]

                                                          │

                                                          ▼

                                                Local Transformer ──► 15-codebook RVQ Tokens

                                                                            │

                                                                            ▼

                                                          VQ Token ──►  [4 Frames Chunks] ──► Codec Decoder ──► 24 kHz Waveform Chunks (320 ms)

```



| Component | ONNX Model | Description |

|-----------|------------|-------------|

| Talker LLM | `talker_model_*.onnx` | Qwen3-based talker LM mapping interleaved text+audio tokens embeddings to hidden states and VQ. Maintains a growing KV-cache across the entire generation. |

| Local Talker | `talker_local_model_*.onnx` | Depth-wise decoder generating 15 RVQ codebook entries per frame from talker hidden states and VQ. Creates and discards a fresh KV-cache per frame. |

| LM Head of Local Talker | `talker_local_lm_head.onnx` | Projection head for each of the 15 codebook output of the local talker transformer. |

| Codec Decoder | `codec_decoder_model.onnx` | Decodes VQ+RVQ audio codes back to 24 kHz waveform. Maintains KV-caches and convolutional caches for streaming decode. |

| Speaker Encoder | `speaker_encoder_model.onnx` | ECAPA-TDNN-based speaker encoder. Produces a 1024-dim speaker embedding vector for voice identity cloning. |

| Talker Codec Embed | `talker_codec_embed_model.onnx` | VQ embedding for the talker model. Consists of 2048 token vocabs. |

| Text Embed Projection | `text_embed_proj_model.onnx` | Text embedding and projection for the talker model. Text embedding consists of 151,936 token vocabs. |



## Requirements



```

librosa

numpy

onnxruntime

python-box

soundfile

transformers==4.57.3

```



Example installation with conda env:



```bash

conda create --name qwen3-tts-streaming-onnx-1 python=3.12

conda activate qwen3-tts-streaming-onnx-1

pip install -r requirements.txt

```



## Directory Structure



```

.

├── test_qwen3-tts-streaming_onnx.py        # End-to-end test script

├── README.md

├── requirements.txt

├── qwen3-tts_onnx/  # FP32

│   ├── talker_model_prefill.onnx

│   ├── talker_model_step.onnx

│   ├── talker_local_model_prefill.onnx

│   ├── talker_local_model_step.onnx

│   ├── talker_local_lm_head.onnx

│   ├── codec_decoder_model.onnx

│   ├── speaker_encoder_model.onnx

│   ├── talker_codec_embed_model.onnx

│   └── text_embed_proj_model.onnx

├── configs/

│   ├── config.json                         # Talker, Local Talker, Speaker Encoder config

│   ├── speech_tokenizer_config.json        # Codec config

│   ├── preprocessor_config.json            # Text Processor configs

│   ├── tokenizer_config.json

│   ├── vocab.json

│   └── merges.txt

├── src/

│   ├── core/

│   │   ├── configuration_qwen3_tts.py

│   │   └── processing_qwen3_tts.py

│   ├── inference/

│   │   └── qwen3_tts_inferencer_onnx.py    # Core ONNX inference engine 

│   └── utils/

│       └── audio_utils.py

├── logs/

│   └── <log_synth>.txt

├── audio_ref/

│   └── <reference_speaker>.[wav|mp3|flac]

└── audio_synth/

    └── <synthesized_example>.wav

```



## Usage



### Basic streaming TTS usage



```bash

python -u test_qwen3-tts-streaming_onnx.py >& logs/log_test-streaming-onnx-1.txt

# audio automatically saved in audio_synth/ with default parameters, text, language.

```



### Usage with parameters



- As of 2026/04/27, you can synthesize multiple rounds of text with continuous streaming.



```

python test_qwen3-tts-streaming_onnx.py \

    --onnx_dir qwen3-tts_onnx/ \

    --model_config_path configs/config.json \

    --codec_config_path configs/tokenizer_config.json \

    --preprocessor_config_dir configs/ \

    --temperature 0.85 \

    --top_p 0.8 \

    --top_k 50 \

    --repetition_penalty 1.9 \

    --repetition_window 50 \

    --num_threads 4 \

    --prompt_wav audio_ref/speaker.[wav|flac|mp3] \

    --out_wav output.wav \

    --text "Text to be synthesized" "Yet another text here" "And another" \

    --language "english"

```



### Available Languages

```

"chinese", "english", "german", "italian", "portuguese",

"spanish", "japanese", "korean", "french", "russian"

```



### Programmatic Usage



```python

from src.inference import Qwen3TTSInferencerONNX



# Create inferencer

inferencer = Qwen3TTSInferencerONNX(

    talker_prefill, talker_step, talker_local_prefill, talker_local_step,

    talker_local_lm_head, codec_decoder,

    speaker_encoder, talker_codec_embed, text_embed_proj,

    preprocessor_config_dir, model_config, codec_config,

    audio_ref_path, language,

)

inferencer.reset_turn(reset_cache=True)



# Stream text and collect audio

for delta in your_llm_stream():

    audio_frames = inferencer.push_text(delta)

    ...

    for audio_tokens in audio_frames:

        ...

        inferencer.push_tokens(audio_tokens)

        for wav in inferencer.audio_chunks():

            ...

            yield wav

```



### Command-Line Arguments



| Argument | Type | Default | Description |

|----------|------|---------|-------------|

| `--onnx_dir` | str | "qwen3-tts_onnx/" | Directory path to all onnx models |

| `--preprocessor_config_dir` | str | "configs/" | Directory path to configuration files for the Qwen3 text tokenizer |

| `--model_config_path` | str | "configs/config.json" | Path to original model configuration file for the Qwen3-TTS-12Hz-0.6B-Base |

| `--codec_config_path` | str | "configs/speech_tokenizer_config.json" | Path to original model configuration file for the codec of Qwen3-TTS-12Hz-0.6B-Base |

| `--temperature` | float | `0.85` | Sampling temperature |

| `--top_p` | float | `0.8` | Nucleus sampling threshold |

| `--top_k` | int | `50` | Top-k sampling cutoff |

| `--repetition_penalty` | float | `1.9` | Repetition penalty coefficient |

| `--repetition_window` | int | `50` | Window for repetition penalty |

| `--delta_chunk_chars` | int | `1` | Characters per simulated LLM delta |

| `--delta_delay_s` | float | `0.0` | Delay between simulated deltas (seconds) |

| `--num_threads` | int | `4` | Number of threads used in sess.intra_op_num_threads of the onnxruntime session options |

| `--prompt_wav` | str | audio_ref/female_shadowheart.flac | Reference speaker audio for voice cloning |

| `--out_wav` | str | `out_streaming.wav` | Output WAV file path |

| `--text` | str | *(Russian text)* | Text to synthesize |

| `--language` | str | "russian" | Language of the text to synthesize |



#### By: [Patrick Lumbantobing](https://www.linkedin.com/in/patrick-lumban-tobing)



#### Copyright@[VertoX-AI](https://www.linkedin.com/company/vertoxai/)



### Citation



If you use this system in your research, please cite:



```bibtex

@misc{vertoxai2026qwen3ttsstreamingonnxcudagraph,

  title={Qwen3-TTS-Streaming-ONNX — VertoX-AI},

  author={Tobing, P. L., VertoX-AI},

  year={2026},

  publisher={HuggingFace},

}

```



## License



This project is licensed under the Apache-2.0, the same license as the original Qwen3-TTS.



```

Created by: Patrick Lumbantobing, Vertox-AI

Copyright (c) 2026 Vertox-AI. All rights reserved.



This work is licensed under the Apache License, Version 2.0.

To view a copy of this license, visit [LICENSE](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md).

```



---



## Acknowledgements



- [Qwen3-TTS-12Hz-0.6B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base) for the original Qwen3-TTS model.

- [Qwen3-TTS Technical Report](https://arxiv.org/abs/2601.15621) (Hu et al., 2026).

- [MOSS-TTS-Realtime](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Realtime) for the reference on the streaming engine.

- [ONNX Runtime](https://onnxruntime.ai/) for high-performance cross-platform inference.