README.md

---
library_name: transformers
tags:
- speech
- audio
- multimodal
license: cc-by-nc-4.0
language:
- en
- ko
pipeline_tag: any-to-any
---

# Raon-Speech-9B

<div align="center">
  <img class="block dark:hidden" src="assets/Raon-Speech-Gradient-Black.png" alt="Raon-Speech Logo" width="400">
  <img class="hidden dark:block" src="assets/Raon-Speech-Gradient-White.png" alt="Raon-Speech Logo" width="400">
</div>

<p align="center">
  <a href="https://www.krafton.ai/ko/"><img src="https://img.shields.io/badge/Homepage-KRAFTON%20AI-blue?style=flat&logo=google-chrome&logoColor=white" alt="Homepage"></a>
  <a href="https://github.com/krafton-ai/Raon-Speech"><img src="https://img.shields.io/badge/GitHub-Raon-white?style=flat&logo=github&logoColor=black" alt="GitHub"></a>
  <br>
  <a href="https://huggingface.co/KRAFTON"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-KRAFTON-yellow?style=flat" alt="Hugging Face"></a>
  <a href="https://x.com/Krafton_AI"><img src="https://img.shields.io/badge/X-KRAFTON%20AI-white?style=flat&logo=x&logoColor=black" alt="X"></a>
  <br>
  <a href="https://creativecommons.org/licenses/by-nc/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey?style=flat" alt="License"></a>
</p>

<p align="center">
  Technical Report | Blog (Coming soon)
  <!-- 📄 <a href="https://arxiv.org/abs/YOUR_PAPER_ID">Technical Report</a> | 📝 <a href="https://YOUR_BLOG_URL">Blog</a> -->
</p>

Raon-Speech is a 9B-parameter speech language model that supports state-of-the-art speech understanding, answering and generation in English and Korean.
This model successfully transforms a pre-trained LLM into a SpeechLM to both understand and generate speech without compromising its original language capabilities.
It trains on millions of hours of English-Korean speech-text datasets with the following training stages: (1) speech encoder-decoder alignment, (2) end-to-end SpeechLM pre-training, and (3) multi-reward DPO-based post-training.

## Key Features

- **End-to-End Speech Language Model**: 9B-parameter multimodal model built on Qwen3 (36 layers, 4096 hidden dim), Qwen3OmniMoeAudioEncoder (24 layers), Mimi codec (32 quantizers), and ECAPA-TDNN speaker encoder.
- **Bilingual Support**: State-of-the-art speech understanding, answering, and generation in both English and Korean.
- **Multi-Task Capabilities**: Supports STT (audio → text), TTS (text → audio), TextQA (text + audio → text), and SpeechChat (audio → text) in a single unified model.
- **Speaker Voice Conditioning**: TTS with optional speaker reference audio for voice cloning via ECAPA-TDNN embeddings.
- **TTS Continuation**: Generate speech that naturally continues from a reference audio, with prefill-based continuation for seamless prosody.
- **Multi-Reward DPO Post-Training**: Three-stage training pipeline — (1) speech encoder-decoder alignment, (2) end-to-end SpeechLM pre-training, and (3) multi-reward DPO-based post-training — for high-quality speech generation.
- **HuggingFace Transformers Integration**: Load and run directly via `AutoModel.from_pretrained` with `trust_remote_code=True` — no custom package installation required.

## Benchmark Results

Measured with LibriSpeech test-clean samples on single-GPU setups via streaming TTS. All values are averaged.

| Metric | RTX 6000 Pro | L40S |
|--------|-------------|------|
| **RTF** | 0.27 (3.7× real-time) | 0.45 (2.2× real-time) |
| **TTFT** | 617 ms | 887 ms |
| **TBT** | 135 ms | 233 ms |

- **RTF** (Real-Time Factor): Lower is faster. Values below 1.0 mean faster-than-real-time synthesis.
- **TTFT** (Time to First Token): Latency until the first audio chunk is returned.
- **TBT** (Time Between Tokens): Average interval between consecutive audio chunks.


## Requirements

```bash
pip install transformers>=4.57.1 torch torchaudio soundfile accelerate

# Optional
pip install speechbrain  # for TTS with speaker voice conditioning
pip install gradio       # for Gradio demo
```

## Quick Start

### Option 1: Load from Hub (recommended)

No `pip install raon` needed.

```python
from transformers import AutoConfig
from transformers.dynamic_module_utils import get_class_from_dynamic_module

MODEL_ID = "KRAFTON/Raon-Speech-9B"

_cfg = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=True)
RaonPipeline = get_class_from_dynamic_module(
    "modeling_raon.RaonPipeline",
    MODEL_ID,
    revision=getattr(_cfg, "_commit_hash", None),
)
del _cfg

pipe = RaonPipeline(MODEL_ID, device="cuda", dtype="bfloat16")
```

### Option 2: With raon package installed

```bash
git clone https://github.com/krafton-ai/Raon-Speech.git
cd Raon-Speech/raon
pip install -e .  # or: uv sync
```

```python
from raon import RaonPipeline

# From Hub (local code + Hub weights)
pipe = RaonPipeline("KRAFTON/Raon-Speech-9B")

# From local path
pipe = RaonPipeline("/path/to/raon-model")
```

## Tasks

#### STT (Audio → Text)
```python
text = pipe.stt("audio.wav")
```

#### TTS (Text → Audio)
```python
# Without speaker conditioning
audio, sr = pipe.tts("Hello, how are you?")
pipe.save_audio((audio, sr), "output.wav")

# With speaker conditioning (requires speechbrain)
audio, sr = pipe.tts("Hello, how are you?", speaker_audio="speaker_ref.wav")
```

#### TextQA (Text + Audio → Text)
```python
answer = pipe.textqa("What is the speaker saying?", audio="audio.wav")
```

#### SpeechChat (Audio → Text)
```python
answer = pipe.speech_chat("question.wav")
```

#### Chat (Multimodal)
```python
messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "audio.wav"},
            {"type": "text", "text": "Transcribe and summarise this audio."},
        ],
    },
]
response = pipe.chat(messages)
```

## Deployment (vLLM-Omni)

####

# 1. Clone & Build
```bash
git clone https://github.com/krafton-ai/vllm-omni.git
cd vllm-omni
docker build -f docker/Dockerfile.ci -t vllm-omni .
```

# 2. Serve
```bash
docker run --rm --gpus all \
  --shm-size=16g \
  -p 8000:8000 \
  vllm-omni \
  bash -c "vllm serve KRAFTON/Raon-Speech-9B --omni --port 8000 --trust-remote-code"
```

# 3. Test — TTS
```bash
curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Hello, how are you?",
    "model": "KRAFTON/Raon-Speech-9B",
    "response_format": "wav"
  }' --output output.wav
```

# 4. Test — TTS with voice cloning
```bash
curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Hello, how are you?",
    "model": "KRAFTON/Raon-Speech-9B",
    "ref_audio": "data:audio/wav;base64,'$(base64 -w0 speaker_ref.wav)'",
    "task_type": "Base",
    "response_format": "wav"
  }' --output cloned.wav
```

# 5. Test — STT
```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "KRAFTON/Raon-Speech-9B",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "audio_url", "audio_url": {"url": "data:audio/wav;base64,'"$(base64 -w0 audio.wav)"'"}},
          {"type": "text", "text": "Transcribe the audio into text."}
        ]
      }
    ]
  }'
```


## License

This repository is licensed under the
[Creative Commons Attribution-NonCommercial 4.0 International License](https://creativecommons.org/licenses/by-nc/4.0/).

© 2026 KRAFTON