feat: add Orpheus TTS continuous batching benchmark

- Implement vLLM AsyncLLMEngine for parallel TTS processing
- Achieve 12x capacity improvement (12.6 real-time users vs 1 sequential)
- RTF improved from 0.85 to 0.079 with 16 concurrent requests
- Add stress test script validating 4/8/12/16 simultaneous users

Test results on RTX 4090:
| Requests | RTF | Real-time Users |
|----------|-------|-----------------|
| 4 | 0.253 | 4.0 |
| 8 | 0.129 | 7.8 |
| 12 | 0.092 | 10.9 |
| 16 | 0.079 | 12.6 |

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs/orpheus-tts-benchmark.md

@@ -0,0 +1,84 @@
+# Orpheus TTS - Benchmark com Continuous Batching
+
+**Data:** 2024-12-25
+**GPU:** NVIDIA RTX 4090 (24GB VRAM)
+**Modelo:** canopylabs/orpheus-3b-0.1-ft (3B parametros)
+
+## Resumo
+
+Implementamos e testamos **Continuous Batching** com vLLM AsyncLLMEngine para o Orpheus TTS, conseguindo **12x mais capacidade** comparado ao uso sequencial.
+
+## Resultados do Teste de Stress
+
+| Requests Simultaneos | Taxa Sucesso | Tempo Wall-Clock | Audio Gerado | RTF | Usuarios Real-Time |
+|---------------------|--------------|------------------|--------------|-----|-------------------|
+| 4 | 100% | 2.93s | 11.61s | 0.253 | 4.0 |
+| 8 | 100% | 2.94s | 22.78s | 0.129 | 7.8 |
+| 12 | 100% | 3.13s | 34.05s | 0.092 | 10.9 |
+| 16 | 100% | 4.03s | 50.77s | 0.079 | **12.6** |
+
+## Comparacao
+
+| Metodo | RTF | Usuarios Real-Time | Melhoria |
+|--------|-----|-------------------|----------|
+| Sequencial (orpheus_tts lib) | 0.85 | ~1 | baseline |
+| Continuous Batching (4 req) | 0.253 | 4.0 | 4x |
+| Continuous Batching (16 req) | 0.079 | 12.6 | **12x** |
+
+## Otimizacoes Ativas (vLLM 0.13.0)
+
+- Flash Attention 2
+- Chunked Prefill
+- Prefix Caching
+- CUDA Graphs
+- torch.compile
+
+## Configuracao do Engine
+
+```python
+engine_args = AsyncEngineArgs(
+    model="canopylabs/orpheus-3b-0.1-ft",
+    dtype="bfloat16",
+    max_model_len=4096,
+    gpu_memory_utilization=0.9,
+    max_num_seqs=16,  # Continuous batching
+    enable_chunked_prefill=True,
+    enable_prefix_caching=True,
+    enforce_eager=False,
+)
+```
+
+## Problema com a Biblioteca orpheus_tts
+
+A biblioteca oficial `orpheus_tts` tem um bug onde o engine vLLM morre apos a primeira inferencia (`EngineDeadError`). A solucao e usar o `AsyncLLMEngine` do vLLM diretamente.
+
+## Formato do Prompt Orpheus
+
+```python
+# Tokens especiais
+START_TOKEN = 128259
+END_TOKENS = [128009, 128260, 128261, 128257]
+STOP_TOKEN = 128258
+AUDIO_TOKEN_BASE = 128266
+
+# Formato do prompt
+prompt = f"{voice}: {text}"
+# Depois adiciona os tokens especiais
+```
+
+## Decodificacao de Audio (SNAC)
+
+Os tokens de audio sao organizados em frames de 7 tokens:
+- Layer 0: offset 0
+- Layer 1: offset 4096 (2 tokens por frame)
+- Layer 2: offset 8192, 12288, etc. (4 tokens por frame)
+
+## Arquivos
+
+- `scripts/orpheus_continuous_batching.py` - Script de teste
+- `scripts/orpheus_stress_test.py` - Teste de stress
+- `orpheus_demo_3_phrases.wav` - Audio de demonstracao
+
+## Conclusao
+
+Com Continuous Batching, uma unica RTX 4090 pode suportar **~12-13 usuarios simultaneos em tempo real** para TTS com Orpheus, uma melhoria de **12x** sobre o metodo sequencial.

scripts/orpheus_continuous_batching.py

@@ -0,0 +1,322 @@
+"""
+Orpheus TTS - Continuous Batching Test v3
+==========================================
+
+Uses AsyncLLMEngine with correct token decoding based on Axolotl's preprocessing.
+Token format per 7-token frame:
+  [layer0, layer1_a, layer2_a, layer2_b, layer1_b, layer2_c, layer2_d]
+  Where:
+  - layer0: 128266 + value
+  - layer1_a: 128266 + 4096 + value
+  - layer2_a: 128266 + 2*4096 + value
+  - layer2_b: 128266 + 3*4096 + value
+  - layer1_b: 128266 + 4*4096 + value
+  - layer2_c: 128266 + 5*4096 + value
+  - layer2_d: 128266 + 6*4096 + value
+"""
+
+import os
+import sys
+import time
+import wave
+import asyncio
+import numpy as np
+
+os.environ["VLLM_ATTENTION_BACKEND"] = "FLASH_ATTN"
+
+import torch
+from transformers import AutoTokenizer
+from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
+from snac import SNAC
+
+# Orpheus special tokens
+START_TOKEN = 128259
+END_TOKENS = [128009, 128260, 128261, 128257]
+STOP_TOKEN = 128258
+AUDIO_TOKEN_BASE = 128266
+
+
+def decode_tokens_to_audio(token_ids, snac_model):
+    """Decode Orpheus tokens to audio using SNAC with correct layer offsets."""
+    # Filter audio tokens and decode by layer
+    audio_frames = []
+    
+    for t in token_ids:
+        if isinstance(t, str):
+            continue
+        if t >= AUDIO_TOKEN_BASE:
+            # Determine which layer this token belongs to
+            offset = t - AUDIO_TOKEN_BASE
+            layer = offset // 4096
+            value = offset % 4096
+            audio_frames.append((layer, value))
+    
+    if len(audio_frames) < 7:
+        return None
+    
+    # Group into 7-token frames and extract codes for each layer
+    num_complete_frames = len(audio_frames) // 7
+    if num_complete_frames == 0:
+        return None
+    
+    codes_0 = []  # layer 0
+    codes_1 = []  # layer 1
+    codes_2 = []  # layer 2
+    
+    for i in range(num_complete_frames):
+        base = i * 7
+        # Frame format: [l0, l1_a, l2_a, l2_b, l1_b, l2_c, l2_d]
+        codes_0.append(audio_frames[base][1])      # layer 0 value
+        codes_1.append(audio_frames[base + 1][1])  # layer 1 first
+        codes_1.append(audio_frames[base + 4][1])  # layer 1 second
+        codes_2.append(audio_frames[base + 2][1])  # layer 2 first
+        codes_2.append(audio_frames[base + 3][1])  # layer 2 second
+        codes_2.append(audio_frames[base + 5][1])  # layer 2 third
+        codes_2.append(audio_frames[base + 6][1])  # layer 2 fourth
+    
+    try:
+        # Convert to tensors with correct shape
+        with torch.no_grad():
+            codes = [
+                torch.tensor(codes_0, dtype=torch.int64).unsqueeze(0).to("cuda"),
+                torch.tensor(codes_1, dtype=torch.int64).unsqueeze(0).to("cuda"),
+                torch.tensor(codes_2, dtype=torch.int64).unsqueeze(0).to("cuda"),
+            ]
+            audio = snac_model.decode(codes)
+        
+        return audio.squeeze().cpu().numpy()
+    except Exception as e:
+        print(f"    Decode error: {e}")
+        return None
+
+
+async def run_tests():
+    """Run continuous batching tests."""
+    print("=" * 60)
+    print("ORPHEUS TTS - CONTINUOUS BATCHING TEST v3")
+    print("=" * 60)
+    
+    # Load tokenizer
+    print("\n[1] Loading tokenizer...")
+    tokenizer = AutoTokenizer.from_pretrained("canopylabs/orpheus-3b-0.1-ft")
+    
+    # Create AsyncLLMEngine
+    print("\n[2] Loading vLLM AsyncLLMEngine...")
+    start_load = time.time()
+    
+    engine_args = AsyncEngineArgs(
+        model="canopylabs/orpheus-3b-0.1-ft",
+        dtype="bfloat16",
+        max_model_len=4096,
+        gpu_memory_utilization=0.9,
+        max_num_seqs=8,  # Continuous batching
+        enable_chunked_prefill=True,
+        enable_prefix_caching=True,
+        enforce_eager=False,
+    )
+    
+    engine = AsyncLLMEngine.from_engine_args(engine_args)
+    load_time = time.time() - start_load
+    print(f"    vLLM loaded in {load_time:.2f}s")
+    
+    # Load SNAC decoder
+    print("\n[3] Loading SNAC decoder...")
+    snac = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval().to("cuda")
+    print("    SNAC loaded!")
+    
+    # Sampling params
+    sampling_params = SamplingParams(
+        temperature=0.2,
+        top_p=0.9,
+        max_tokens=4096,
+        stop_token_ids=[STOP_TOKEN],
+        repetition_penalty=1.1,
+    )
+    
+    def format_prompt(text: str, voice: str = "tara") -> str:
+        """Format prompt with Orpheus special tokens."""
+        adapted_prompt = f"{voice}: {text}"
+        prompt_tokens = tokenizer(adapted_prompt, return_tensors="pt")
+        start_token = torch.tensor([[START_TOKEN]], dtype=torch.int64)
+        end_tokens = torch.tensor([END_TOKENS], dtype=torch.int64)
+        all_input_ids = torch.cat([start_token, prompt_tokens.input_ids, end_tokens], dim=1)
+        prompt_string = tokenizer.decode(all_input_ids[0])
+        return prompt_string
+    
+    async def generate_speech(text: str, voice: str = "tara", request_id: str = None):
+        """Generate speech for a single request."""
+        prompt_string = format_prompt(text, voice)
+        request_id = request_id or f"req_{time.time()}"
+        
+        start = time.time()
+        token_ids = []
+        
+        async for output in engine.generate(prompt_string, sampling_params, request_id):
+            token_ids = list(output.outputs[0].token_ids)
+        
+        gen_time = time.time() - start
+        
+        # Count audio tokens
+        audio_token_count = sum(1 for t in token_ids if isinstance(t, int) and t >= AUDIO_TOKEN_BASE)
+        
+        # Decode to audio
+        audio = decode_tokens_to_audio(token_ids, snac)
+        
+        if audio is None:
+            return {
+                'success': False,
+                'text': text[:30],
+                'gen_time': gen_time,
+                'tokens': len(token_ids),
+                'audio_tokens': audio_token_count,
+            }
+        
+        audio_duration = len(audio) / 24000
+        rtf = gen_time / audio_duration if audio_duration > 0 else float('inf')
+        
+        return {
+            'success': True,
+            'text': text[:30],
+            'gen_time': gen_time,
+            'tokens': len(token_ids),
+            'audio_tokens': audio_token_count,
+            'audio_duration': audio_duration,
+            'rtf': rtf,
+            'audio': audio,
+        }
+    
+    # =========================================================================
+    # TEST 1: SEQUENTIAL (baseline)
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("[4] Test SEQUENTIAL (baseline)")
+    print("=" * 60)
+    
+    test_texts = [
+        "Hello! This is the first test.",
+        "Second test to measure performance.",
+        "Third test for consistent results.",
+    ]
+    
+    sequential_results = []
+    total_seq_time = 0
+    
+    for i, text in enumerate(test_texts, 1):
+        print(f"\n    Test {i}: \"{text}\"")
+        result = await generate_speech(text, request_id=f"seq_{i}")
+        
+        if result['success']:
+            print(f"    -> Time: {result['gen_time']:.2f}s | Audio: {result['audio_duration']:.2f}s | RTF: {result['rtf']:.3f}")
+            sequential_results.append(result)
+            total_seq_time += result['gen_time']
+        else:
+            print(f"    -> ERROR: {result['tokens']} tokens ({result['audio_tokens']} audio), no audio")
+    
+    # =========================================================================
+    # TEST 2: PARALLEL (Continuous Batching)
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("[5] Test PARALLEL (Continuous Batching)")
+    print("=" * 60)
+    
+    parallel_texts = [
+        "Hello, how are you today?",
+        "The weather is beautiful outside.",
+        "I love programming with Python.",
+        "Machine learning is fascinating.",
+    ]
+    
+    batch_results_summary = []
+    
+    for num_concurrent in [2, 4]:
+        print(f"\n    === {num_concurrent} CONCURRENT REQUESTS ===")
+        
+        texts = parallel_texts[:num_concurrent]
+        
+        start_batch = time.time()
+        tasks = [
+            generate_speech(text, request_id=f"par_{num_concurrent}_{i}")
+            for i, text in enumerate(texts)
+        ]
+        results = await asyncio.gather(*tasks)
+        batch_time = time.time() - start_batch
+        
+        # Calculate metrics
+        successful = [r for r in results if r['success']]
+        total_audio = sum(r['audio_duration'] for r in successful)
+        
+        print(f"\n    Results:")
+        for r in results:
+            if r['success']:
+                print(f"    - \"{r['text']}...\" -> {r['gen_time']:.2f}s | {r['audio_duration']:.2f}s | RTF: {r['rtf']:.3f}")
+            else:
+                print(f"    - \"{r['text']}...\" -> FAILED ({r['audio_tokens']} audio tokens)")
+        
+        if total_audio > 0:
+            batch_rtf = batch_time / total_audio
+            throughput = len(successful) / batch_time
+            
+            print(f"\n    Aggregate Metrics:")
+            print(f"    - Total wall-clock time: {batch_time:.2f}s")
+            print(f"    - Success rate: {len(successful)}/{len(texts)}")
+            print(f"    - Total audio generated: {total_audio:.2f}s")
+            print(f"    - Batch RTF: {batch_rtf:.3f}")
+            print(f"    - Throughput: {throughput:.2f} req/s")
+            print(f"    - Effective speed: {1/batch_rtf:.1f}x real-time")
+            
+            batch_results_summary.append({
+                'concurrent': num_concurrent,
+                'batch_time': batch_time,
+                'total_audio': total_audio,
+                'batch_rtf': batch_rtf,
+                'throughput': throughput,
+            })
+    
+    # =========================================================================
+    # FINAL SUMMARY
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("[6] === FINAL SUMMARY ===")
+    print("=" * 60)
+    
+    if sequential_results:
+        seq_rtfs = [r['rtf'] for r in sequential_results]
+        avg_seq_rtf = sum(seq_rtfs) / len(seq_rtfs)
+        total_seq_audio = sum(r['audio_duration'] for r in sequential_results)
+        
+        print(f"\n    SEQUENTIAL (baseline):")
+        print(f"    - Average RTF: {avg_seq_rtf:.3f}")
+        print(f"    - Speed: {1/avg_seq_rtf:.1f}x real-time")
+        
+        if batch_results_summary:
+            print(f"\n    CONTINUOUS BATCHING:")
+            for bs in batch_results_summary:
+                speedup = avg_seq_rtf / bs['batch_rtf'] if bs['batch_rtf'] > 0 else 0
+                print(f"    - {bs['concurrent']} concurrent: RTF={bs['batch_rtf']:.3f}, {bs['throughput']:.2f} req/s, {speedup:.1f}x speedup")
+        
+        # Capacity estimate
+        users_seq = 1 / avg_seq_rtf if avg_seq_rtf > 0 else 0
+        print(f"\n    RTX 4090 CAPACITY ESTIMATE:")
+        print(f"    - Sequential: ~{users_seq:.0f} real-time users")
+        if batch_results_summary:
+            best_batch = min(batch_results_summary, key=lambda x: x['batch_rtf'])
+            users_batch = best_batch['concurrent'] / best_batch['batch_rtf'] if best_batch['batch_rtf'] > 0 else 0
+            print(f"    - With batching: ~{users_batch:.0f} real-time users")
+    
+    print("=" * 60)
+    
+    # Save audio
+    if sequential_results and sequential_results[-1]['success']:
+        audio = sequential_results[-1]['audio']
+        output_path = "/root/test_batching_v3_output.wav"
+        audio_int16 = (audio * 32767).astype(np.int16)
+        with wave.open(output_path, "wb") as wf:
+            wf.setnchannels(1)
+            wf.setsampwidth(2)
+            wf.setframerate(24000)
+            wf.writeframes(audio_int16.tobytes())
+        print(f"\n    Audio saved to: {output_path}")
+
+
+if __name__ == "__main__":
+    asyncio.run(run_tests())

scripts/orpheus_stress_test.py

@@ -0,0 +1,195 @@
+"""
+Teste de Stress - Validar capacidade real de usuarios simultaneos
+"""
+import os
+os.environ["VLLM_ATTENTION_BACKEND"] = "FLASH_ATTN"
+
+import torch
+import time
+import asyncio
+import numpy as np
+from transformers import AutoTokenizer
+from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams
+from snac import SNAC
+
+START_TOKEN = 128259
+END_TOKENS = [128009, 128260, 128261, 128257]
+STOP_TOKEN = 128258
+AUDIO_TOKEN_BASE = 128266
+
+def decode_tokens_to_audio(token_ids, snac_model):
+    audio_frames = []
+    for t in token_ids:
+        if isinstance(t, str):
+            continue
+        if t >= AUDIO_TOKEN_BASE:
+            offset = t - AUDIO_TOKEN_BASE
+            layer = offset // 4096
+            value = offset % 4096
+            audio_frames.append((layer, value))
+    
+    if len(audio_frames) < 7:
+        return None
+    
+    num_complete_frames = len(audio_frames) // 7
+    if num_complete_frames == 0:
+        return None
+    
+    codes_0, codes_1, codes_2 = [], [], []
+    for i in range(num_complete_frames):
+        base = i * 7
+        codes_0.append(audio_frames[base][1])
+        codes_1.append(audio_frames[base + 1][1])
+        codes_1.append(audio_frames[base + 4][1])
+        codes_2.append(audio_frames[base + 2][1])
+        codes_2.append(audio_frames[base + 3][1])
+        codes_2.append(audio_frames[base + 5][1])
+        codes_2.append(audio_frames[base + 6][1])
+    
+    try:
+        with torch.no_grad():
+            codes = [
+                torch.tensor(codes_0, dtype=torch.int64).unsqueeze(0).to("cuda"),
+                torch.tensor(codes_1, dtype=torch.int64).unsqueeze(0).to("cuda"),
+                torch.tensor(codes_2, dtype=torch.int64).unsqueeze(0).to("cuda"),
+            ]
+            audio = snac_model.decode(codes)
+        return audio.squeeze().cpu().numpy()
+    except:
+        return None
+
+async def main():
+    print("=" * 70)
+    print("TESTE DE STRESS - VALIDAR CAPACIDADE REAL DE USUARIOS")
+    print("=" * 70)
+    
+    print("\n[1] Carregando modelo...")
+    tokenizer = AutoTokenizer.from_pretrained("canopylabs/orpheus-3b-0.1-ft")
+    
+    engine_args = AsyncEngineArgs(
+        model="canopylabs/orpheus-3b-0.1-ft",
+        dtype="bfloat16",
+        max_model_len=4096,
+        gpu_memory_utilization=0.9,
+        max_num_seqs=16,
+        enable_chunked_prefill=True,
+        enable_prefix_caching=True,
+        enforce_eager=False,
+    )
+    engine = AsyncLLMEngine.from_engine_args(engine_args)
+    snac = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval().to("cuda")
+    
+    sampling_params = SamplingParams(
+        temperature=0.2,
+        top_p=0.9,
+        max_tokens=4096,
+        stop_token_ids=[STOP_TOKEN],
+        repetition_penalty=1.1,
+    )
+    
+    def format_prompt(text, voice="tara"):
+        adapted_prompt = f"{voice}: {text}"
+        prompt_tokens = tokenizer(adapted_prompt, return_tensors="pt")
+        start_token = torch.tensor([[START_TOKEN]], dtype=torch.int64)
+        end_tokens = torch.tensor([END_TOKENS], dtype=torch.int64)
+        all_input_ids = torch.cat([start_token, prompt_tokens.input_ids, end_tokens], dim=1)
+        return tokenizer.decode(all_input_ids[0])
+    
+    async def generate_speech(text, request_id):
+        prompt_string = format_prompt(text)
+        start = time.time()
+        token_ids = []
+        async for output in engine.generate(prompt_string, sampling_params, request_id):
+            token_ids = list(output.outputs[0].token_ids)
+        gen_time = time.time() - start
+        
+        audio = decode_tokens_to_audio(token_ids, snac)
+        if audio is None:
+            return {'success': False, 'gen_time': gen_time}
+        
+        audio_duration = len(audio) / 24000
+        return {
+            'success': True,
+            'gen_time': gen_time,
+            'audio_duration': audio_duration,
+            'rtf': gen_time / audio_duration if audio_duration > 0 else float('inf')
+        }
+    
+    test_texts = [
+        "Hello, how are you doing today?",
+        "The weather is beautiful outside.",
+        "I love programming with Python.",
+        "Machine learning is fascinating.",
+        "Can you help me with this task?",
+        "Let me explain how this works.",
+        "This is a test of the system.",
+        "Technology is amazing these days.",
+        "Have a wonderful day ahead.",
+        "Thank you for your patience.",
+        "Let's work together on this.",
+        "The future looks very bright.",
+        "I appreciate your help today.",
+        "This demonstration is working.",
+        "Audio generation is fast now.",
+        "Real-time speech synthesis.",
+    ]
+    
+    print("\n[2] Iniciando testes de stress...")
+    print("=" * 70)
+    
+    results_summary = []
+    
+    for num_users in [4, 8, 12, 16]:
+        print(f"\n>>> TESTANDO {num_users} USUARIOS SIMULTANEOS <<<")
+        print("-" * 50)
+        
+        texts = test_texts[:num_users]
+        
+        start_batch = time.time()
+        tasks = [generate_speech(text, f"user_{i}_{num_users}") for i, text in enumerate(texts)]
+        results = await asyncio.gather(*tasks)
+        batch_time = time.time() - start_batch
+        
+        successful = [r for r in results if r['success']]
+        total_audio = sum(r['audio_duration'] for r in successful)
+        
+        if len(successful) > 0:
+            batch_rtf = batch_time / total_audio
+            throughput = len(successful) / batch_time
+            realtime_users = total_audio / batch_time
+            
+            print(f"    Sucesso: {len(successful)}/{num_users}")
+            print(f"    Tempo total (wall-clock): {batch_time:.2f}s")
+            print(f"    Audio total gerado: {total_audio:.2f}s")
+            print(f"    Batch RTF: {batch_rtf:.3f}")
+            print(f"    Throughput: {throughput:.2f} req/s")
+            print(f"    USUARIOS REAL-TIME: {realtime_users:.1f}")
+            
+            results_summary.append({
+                'users': num_users,
+                'success': len(successful),
+                'batch_time': batch_time,
+                'total_audio': total_audio,
+                'batch_rtf': batch_rtf,
+                'realtime_users': realtime_users
+            })
+        else:
+            print(f"    ERRO: Nenhum audio gerado!")
+    
+    print("\n" + "=" * 70)
+    print("RESUMO FINAL - CAPACIDADE RTX 4090")
+    print("=" * 70)
+    
+    print("\n| Requests | Sucesso | Wall-Time | Audio Total | RTF   | Real-time Users |")
+    print("|----------|---------|-----------|-------------|-------|-----------------|")
+    for r in results_summary:
+        print(f"| {r['users']:8} | {r['success']:7} | {r['batch_time']:9.2f}s | {r['total_audio']:11.2f}s | {r['batch_rtf']:.3f} | {r['realtime_users']:15.1f} |")
+    
+    if results_summary:
+        best = max(results_summary, key=lambda x: x['realtime_users'])
+        print(f"\nMELHOR RESULTADO: {best['realtime_users']:.1f} usuarios real-time com {best['users']} requests")
+    
+    print("=" * 70)
+
+if __name__ == "__main__":
+    asyncio.run(main())