---
license: apache-2.0
tags:
- audio
- speech
- audio-to-audio
- speech-language-models
datasets:
- amphion/Emilia-Dataset
- facebook/multilingual_librispeech
- CSTR-Edinburgh/vctk
- google/fleurs
- mozilla-foundation/common_voice_13_0
- mythicinfinity/libritts_r
extra_gated_fields:
  Affiliation (Company or Academic Institution): text
  Role: text
---

# NeuCodec 🎧

[![NeuCodec Intro](NeuCodec-Thumbnail.jpg)](https://www.youtube.com/watch?v=O7XH1lGZyYY)

*Click the image above to see NeuCodec in action on Youtube!*

*Created by Neuphonic - building faster, smaller, on-device voice AI*

A lightweight neural codec that encodes audio at just 0.8 kbps - perfect for researchers and builders who need something that *just works* for training high quality text-to-speech models.

# Key Features

* 🔊 Low bit-rate compression - a speech codec that compresses and reconstructs audio with near-inaudible reconstruction loss
<br>
* 🎼 Upsamples from 16kHz → 24kHz
<br>
* 🌍 Ready for real-world use - train your own SpeechLMs without needing to build your own codec
<br>
* 🏢 Commercial use permitted - use it in your own tools or products
<br>
* 📊 Released with large pre-encoded datasets - we’ve compressed Emilia-YODAS from 1.7TB to 41GB using NeuCodec, significantly reducing the compute requirements needed for training 
<br>

# Model Details

NeuCodec is a Finite Scalar Quantisation (FSQ) based 0.8kbps audio codec for speech tokenization.
It takes advantage of the following features:

* FSQ quantisation resulting in a single codebook, making it ideal for downstream modeling with Speech Language Models.
* Trained with CC data such that there are no Non-Commercial data restrictions.
* At 50 tokens/sec and 16 bits per token, the overall bit-rate is 0.8kbps.
* The codec takes in 16kHz input and outputs 24kHz using an upsampling decoder.
* The FSQ encoding scheme allows for bit-level error resistance suitable for unreliable and noisy channels.

NeuCodec is largely based on extending the work of [X-Codec2.0](https://huggingface.co/HKUSTAudio/xcodec2).

- **Developed by:** Neuphonic
- **Model type:** Neural Audio Codec
- **License:** apache-2.0
- **Repository:** https://github.com/neuphonic/neucodec
- **Paper:** [arXiv](https://arxiv.org/abs/2509.09550)
- **Pre-encoded Datasets:**
  - [Emilia-YODAS-EN](https://huggingface.co/datasets/neuphonic/emilia-yodas-english-neucodec)
  - *More coming soon!*

# Setup
NeuCodec is now supported natively in 🤗 Transformers. Until it is part of an official Transformers release, install from source:

```
pip install git+https://github.com/huggingface/transformers
```

## Usage example
Here is a quick example of how to encode and decode an audio using this model:
```python
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor, AutoModel

model_id = "neuphonic/neucodec"
model = AutoModel.from_pretrained(model_id, device_map="auto")
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
dataset = dataset.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate))
audio = dataset[0]["audio"]["array"]
inputs = feature_extractor(audio=audio, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to(
    model.device, model.dtype
)
print("Input waveform shape:", inputs["input_values"].shape)
# Input waveform shape: torch.Size([1, 1, 93760])

# encoder and decoder
audio_codes = model.encode(**inputs).audio_codes
print("Audio codes shape:", audio_codes.shape)
# Audio codes shape: torch.Size([1, 1, 293])
audio_values = model.decode(audio_codes).audio_values
print("Audio values shape:", audio_values.shape)
# Audio values shape: torch.Size([1, 1, 93760])

# Equivalently, you can do encoding and decoding in one step
model_output = model(**inputs)
audio_codes = model_output.audio_codes
audio_values = model_output.audio_values
```

# Get Started (Legacy)

Use the code below to get started with the model.

To install from pypi in a dedicated environment, using Python 3.10 or above:

```bash
conda create -n neucodec python=3.10
conda activate neucodec
pip install neucodec
```
Then, to use in python:

```python
import librosa
import torch
import torchaudio
from torchaudio import transforms as T
from neucodec import NeuCodec
 
model = NeuCodec.from_pretrained("neuphonic/neucodec")
model.eval().cuda()   
 
y, sr = torchaudio.load(librosa.ex("libri1"))
if sr != 16_000:
    y = T.Resample(sr, 16_000)(y)[None, ...] # (B, 1, T_16)

with torch.no_grad():
    fsq_codes = model.encode_code(y)
    # fsq_codes = model.encode_code(librosa.ex("libri1")) # or directly pass your filepath!
    print(f"Codes shape: {fsq_codes.shape}")  
    recon = model.decode_code(fsq_codes).cpu() # (B, 1, T_24)

torchaudio.save("reconstructed.wav", recon[0, :, :], 24_000)
```

# Training Details

The model was trained using the following data: 
* Emilia-YODAS
* MLS
* LibriTTS
* Fleurs
* CommonVoice
* HUI
* Additional proprietary set

All publically available data was covered by either the CC-BY-4.0 or CC0 license.