---
library_name: transformers
pipeline_tag: text-generation
license: apache-2.0
base_model:
  - hypaai/Hypa-Orpheus-3b-TTS-VC
datasets:
  - hypaai/Hypa-Voices-snac
language:
  - en
  - fr
  - es
  - pt
  - ar
  - sw
  - pcm
  - yo
  - ha
  - ig
  - ibb
  - tiv
  - efi
  - anw
  - idc
  - igl
  - nup
  - ego
  - fon
  - urh
  - tw
  - ibr
tags:
  - orpheus
  - llama
  - text-to-speech
  - voice-cloning
  - speech-synthesis
  - snac
  - multilingual
  - low-resource
  - underrepresented-languages
  - unsloth
  - lora
  - qlora
  - continued-finetuning
  - hypa-ai
---

<div align="center">

![Hypa-Orpheus 3B](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC/resolve/main/Hypa_Orpheus.jpg)

**Extended continued fine-tune of Hypa-Orpheus 3B with a higher-capacity LoRA adapter (r=2048), merged to 16-bit bf16 for inference.**

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Base: Hypa-Orpheus 3B](https://img.shields.io/badge/Base-Hypa--Orpheus%203B-blue)](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC)
[![Collection: Hypa-Orpheus](https://img.shields.io/badge/%F0%9F%A4%97%20Collection-Hypa--Orpheus-yellow)](https://huggingface.co/collections/hypaai/hypa-orpheus)
[![Primary release](https://img.shields.io/badge/Primary-Hypa--Orpheus%203B-green)](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC)
[![Trained with Unsloth](https://img.shields.io/badge/Trained%20with-Unsloth-orange)](https://github.com/unslothai/unsloth)

</div>

## Model Description

**Hypa-Orpheus 3B Extended** (`hypaai/Hypa-Orpheus-3b-TTS-VC-ext`) is a **continued fine-tune** of the production [Hypa-Orpheus 3B](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC) release. It uses the same Hypa-Voices corpus, prompt formats, and SNAC training pipeline as the primary model, but attaches a **much higher-rank LoRA adapter** (`r=2048`, `α=2048`) on top of the merged Step **3a** checkpoint and trains further on the same precomputed integer SNAC shards.

This is an **experimental capacity extension**, not a replacement for the primary release. It exists for researchers and deployers who want the strongest held-out loss we observed after continuing training with a larger adapter.

**Collection:** [hypaai/hypa-orpheus](https://huggingface.co/collections/hypaai/hypa-orpheus)

| Repository | Role | Best for |
| --- | --- | --- |
| [`hypaai/Hypa-Orpheus-3b-TTS-VC`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC) | Primary merged release (r=512, step 44,334) | Default production deployment |
| **`hypaai/Hypa-Orpheus-3b-TTS-VC-ext` (this repo)** | Extended merged release (r=2048, step 4,992) | Higher-capacity continuation checkpoint |
| [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs) | Primary LoRA + TensorBoard | Primary adapter inspection and retraining |
| [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs-ext`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs-ext) | Extended LoRA + TensorBoard | Extended adapter inspection and retraining |

| Property | Primary release | **Extended (this repo)** |
| --- | --- | --- |
| **Starting checkpoint** | `unsloth/orpheus-3b-0.1-ft` | `hypaai/Hypa-Orpheus-3b-TTS-VC` |
| **LoRA rank / alpha** | r=512, α=512 | **r=2048, α=2048** |
| **Trainable parameters** | 778M (19.07%) | **~3.11B (~76%)** |
| **Adapter size** | ~3.1 GB | **~12.4 GB** |
| **Effective batch size** | 96 | **256** |
| **Optimizer steps (shipped)** | 44,334 | **4,992** |
| **Best validation loss** | 3.513 | **3.438** |
| **Approx. validation PPL** | 33.5 | **31.1** |

Capabilities, languages, prompt structure, SNAC decode path, and intended use cases are identical to the primary model. See the main model card for the full description:

**[hypaai/Hypa-Orpheus-3b-TTS-VC README](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC)**

---

## What Changed in This Run

The extended run kept the same data mixture and response-only SNAC supervision, but increased adapter capacity and batch size while continuing from the already merged production weights:

| Hyperparameter | Primary release | Extended run |
| --- | --- | --- |
| Base weights | 4-bit QLoRA on Orpheus FT | **Merged bf16 Hypa-Orpheus 3B** |
| LoRA rank (r) | 512 | **2048** |
| LoRA alpha (α) | 512 | **2048** |
| Per-device batch size | 16 | 16 |
| Gradient accumulation | 6 | **16** |
| **Effective batch size** | **96** | **256** |
| Learning rate | 5e-5 | 5e-5 |
| LR schedule | cosine, 3% warmup | cosine, **498 warmup steps** |
| Optimizer | AdamW 8-bit | AdamW 8-bit |
| Weight decay | 0.01 | 0.01 |
| Max grad norm | 1.0 | 1.0 |
| Eval / save interval | 2,217 steps | **832 steps** |
| Planned step budget | 44,334 | **16,626** |
| **Shipped checkpoint** | step 44,334 | **step 4,992** |
| Precision | bfloat16 | bfloat16 |
| Random seed | 3407 | 3407 |

The published extended checkpoint reflects **4,992 optimizer steps**, about **30% of one effective epoch** on the same interleaved TTS/VC mixture. Training was stopped and released at the validation-best step observed in that continuation segment.

---

## Evaluation and Checkpoint Selection

TensorBoard logs for the extended adapter:

**[Hypa-Orpheus 3B Extended LoRA training logs](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs-ext/tensorboard)**

Headline metrics from the shipped continuation:

| Metric | Value | Step |
| --- | --- | --- |
| Initial training loss | **3.582** | 1 |
| Final training loss | **3.433** | 4,992 |
| Initial validation loss | **3.496** | 832 |
| **Best validation loss (shipped)** | **3.438** | **4,992** |

Compared with the primary release validation loss of **3.513** at step **44,334**, the extended checkpoint improves held-out loss by **0.075** (approximate validation perplexity **33.5 → 31.1**). Validation loss improved at every logged evaluation interval in this continuation.

| Step | Training Loss | Validation Loss | Val PPL | Gen. Gap |
| --- | --- | --- | --- | --- |
| 832 | 3.531 | 3.496 | 33.0 | −0.035 |
| 1,664 | 3.491 | 3.478 | 32.4 | −0.013 |
| 2,496 | 3.483 | 3.464 | 31.9 | −0.019 |
| 3,328 | 3.469 | 3.456 | 31.7 | −0.012 |
| 4,160 | 3.463 | 3.451 | 31.5 | −0.012 |
| **4,992** | **3.433** | **3.438** | **31.1** | **+0.006** |

As with the primary release, these scalars measure uncertainty over the supervised target SNAC stream. They do not, by themselves, prove better perceptual quality. The extended checkpoint should be treated as a **loss-improved continuation** pending formal listening and task benchmarks.

Gradient norms remained stable throughout the run (roughly **0.18 → 0.11**).

---

## Which Checkpoint Should I Use?

| Use case | Recommendation |
| --- | --- |
| **Default production** | [`hypaai/Hypa-Orpheus-3b-TTS-VC`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC) |
| **Smaller adapter / full primary training budget** | Primary LoRA repo |
| **Lowest published held-out loss after continuation** | **This repo (`-ext`)** |
| **Further high-rank experimentation** | Extended LoRA repo |

Choose the primary release when you want the fully budgeted Step **3a** training run and the smaller r=512 adapter footprint. Choose this extended release when you want the best validation loss we published after continuing from that checkpoint with a r=2048 adapter, and can accept the larger merged weights and shorter continuation schedule.

---

## Quick Start

### Load the extended merged model

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "hypaai/Hypa-Orpheus-3b-TTS-VC-ext"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
```

Prompt assembly, SNAC encode/decode, and vLLM serving follow the same conventions as the primary release. This repository includes the same `handler.py` inference entrypoint pattern as [`hypaai/Hypa-Orpheus-3b-TTS-VC`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC).

```bash
vllm serve hypaai/Hypa-Orpheus-3b-TTS-VC-ext --dtype bfloat16 --max-model-len 4096
```

### Load the extended LoRA adapter

See [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs-ext`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs-ext) for adapter loading, merging, and TensorBoard links.

---

## Limitations (Extended Release)

- **Shorter continuation budget.** The shipped extended checkpoint covers **4,992** steps, not the full planned **16,626**-step schedule.
- **Much larger adapter.** r=2048 increases adapter storage and merge size roughly **4×** versus the primary release.
- **Experimental status.** The primary release remains our default production artifact; this continuation is published for transparency and research comparison.
- **Same perceptual eval gap.** Improved held-out loss does not yet come with published WER/CER, speaker-similarity, or MOS results.

For shared limitations on languages, prompt sensitivity, and context length, see the [primary model card](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC).

---

## Released Artifacts

- **Extended merged 16-bit model (this repo):** [`hypaai/Hypa-Orpheus-3b-TTS-VC-ext`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-ext)
- **Extended LoRA + TensorBoard:** [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs-ext`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs-ext)
- **Primary merged release:** [`hypaai/Hypa-Orpheus-3b-TTS-VC`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC)
- **Primary LoRA + TensorBoard:** [`hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs`](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-LoRAs)
- **Training data (public subset):** [Hypa-Voices collection](https://huggingface.co/collections/hypaai/hypa-voices)

For questions or contributions, contact **chris@hypaintelligence.com** or open an issue on Hugging Face.

---

## Citation

If you use the extended Hypa-Orpheus checkpoint, please cite both the primary release and this continuation:

```bibtex
@misc{hypaorpheus2026,
  title        = {Hypa-Orpheus 3B: Multilingual Text-to-Speech and Voice Cloning for Underrepresented Languages},
  author       = {{Hypa Intelligence and AfroVoices}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC}}
}

@misc{hypaorpheus2026ext,
  title        = {Hypa-Orpheus 3B Extended: Continued Fine-Tuning with Higher-Capacity LoRA},
  author       = {{Hypa Intelligence and AfroVoices}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC-ext}}
}
```

---

## License

Released under the **Apache License 2.0**.

---

## Acknowledgments

Same upstream acknowledgments as the [primary Hypa-Orpheus release](https://huggingface.co/hypaai/Hypa-Orpheus-3b-TTS-VC#acknowledgments): Canopy Labs (Orpheus), Meta AI (Llama), Hubert Siuzdak (SNAC), Unsloth, RunPod, Mozilla Common Voice, AfroVoices, and Hypa Intelligence Research (HaIR).

---

**Hypa Intelligence** • [Website](https://hypaintelligence.com) • [Hugging Face](https://huggingface.co/hypaai) • [Updates](https://hypaintelligence.com/updates)