| --- |
| |
| |
| license: cc-by-4.0 |
| language: |
| - fr |
| - en |
| library_name: hibiki |
| tags: |
| - speech |
| - translation |
| - streaming |
| metrics: |
| - bleu |
| --- |
| |
| # Model Card for Hibiki |
|
|
| [Hibiki](https://github.com/kyutai-labs/hibiki) is a model for streaming speech translation (also known as *simultaneous* translation). Unlike offline translation—where one waits for the end of the source utterance to start translating--- Hibiki adapts its flow to accumulate just enough context to produce a correct translation in real-time, chunk by chunk. As the user speaks, Hibiki generates natural speech in the target language, optionally with voice transfer, along with a text translation. |
| Hibiki currently only supports French-to-English translation. |
|
|
| ## Model Details |
|
|
| This is a model referred to as *Hibiki-M* (for *Mobile*) in our [paper](https://arxiv.org/abs/2502.03382), a 1.7B parameter |
| hierarchical Transformer producing speech and text tokens at a framerate of 12.5Hz, with audio being generated at a |
| 1.1kbps bitrate. |
|
|
| ### Model Description |
|
|
| Hibiki is a decoder-only model for simultaneous speech translation. Hibiki leverages the multistream architecture of [Moshi](https://arxiv.org/abs/2410.00037) |
| to model source and target speech jointly. This allows Hibiki to continuously process the input stream while generating |
| the target speech. Hibiki produces text and audio tokens at a constant framerate of 12.5Hz. This allows for a continuous |
| output audio stream, along with timestamped text tranlsation. Since Hibiki relies on simple temperature sampling, |
| it is compatible with batching unlike models that rely on complex inference policies. Moreover, the fidelity of Hibiki's |
| voice transfer can be controlled by changing the coefficient of the Classifier-Free Guidance: a larger coefficient will |
| increase voice similarity, but excessive coefficients can lead to worse translations. |
|
|
|
|
| - **Developed by:** Kyutai |
| - **Model type:** Simultaneous speech-to-speech and speech-to-text translation. |
| - **Language(s) (NLP):** French-to-English |
| - **License:** CC-BY |
|
|
| ### Model Sources |
|
|
|
|
| - **Repository:** [repo](https://github.com/kyutai-labs/hibiki) |
| - **Paper:** [paper](https://arxiv.org/abs/2502.03382) |
| - **Examples:** [demo](https://hf.co/spaces/kyutai/hibiki-samples) |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| The model can be used for streaming translation from French to English in real-time settings, or for batched |
| simultaneous translation of many input sequences. It is robust to noisy conditions and is trained on sequences up |
| to 120 seconds. |
|
|
|
|
| ### Downstream Use |
|
|
| Some components of the model can be used independently or repurposed relatively easily. |
| For instance the Mimi codec is a state-of-the-art audio neural codec that combines semantic and acoustic information into audio tokens running at 12Hz and a bitrate of 1.1kbps, which make it particularly adapted to train speech language models or text-to-speech systems. Regarding the main Hibiki architecture, |
| supporting other pairs of languages would require finetuning. |
|
|
|
|
| ### Out-of-Scope Use |
|
|
| The model is not intended to be used to impersonate other people or any malicious use of any kind. |
|
|
|
|
| ## How to Get Started with the Model |
|
|
| See the main [README](https://github.com/kyutai-labs/hibiki) file. |
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| - Textual data: The underlying [Helium](https://huggingface.co/kyutai/helium-1-preview-2b) model is trained on a mix of |
| data including: Wikipedia, Stack Exchange, open-access scientific articles (from peS2o) and Common Crawl. |
|
|
| - Audio data |
|
|
| - **Unsupervised audio dataset:** used for pre-training, this is a collection of 7M hours of readily available audio content in English and 450k hours in French, following the preprocessing and recipe of [Moshi](https://arxiv.org/abs/2410.00037). |
| - **Synthetic translation dataset**: Around 40k hours of parallel French-English data synthesized with *contextual alignment* (see [Section 3.2](https://arxiv.org/pdf/2502.03382)) with various levels of speaker similarity. |
| - **Translation finetuning:** A 900 hours mixture of a resynthesized version of [CVSS-T](https://github.com/google-research-datasets/cvss) and synthetic long-form utterances. |
|
|
| ### Training procedure and hyper-parameters |
|
|
| The different stages of the training procedure are detailled in the paper along with the hyper-parameters. |
|
|
| ### Compute Infrastructure |
|
|
| The final model was trained on 48 H100 Nvidia GPUs. |
|
|
| ## Citation |
|
|
| ``` |
| @misc{labiausse2025hibiki, |
| title={High-Fidelity Simultaneous Speech-To-Speech Translation}, |
| author={Tom Labiausse and Laurent Mazaré and Edouard Grave and Patrick Pérez and Alexandre Défossez and Neil Zeghidour}, |
| year={2025}, |
| eprint={2502.03382}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2502.03382}, |
| } |
| ``` |
|
|
|
|
| ## Model Card Authors |
|
|
| Tom Labiausse, Laurent Mazaré, Edouard Grave, Patrick Pérez, Alexandre Défossez, Neil Zeghidour |
