Spaces:
Sleeping
Streaming Vocos: Neural vocoder for fast streaming applications
Vocos was proposed as fast neural vocoder designed to synthesize audio waveforms from acoustic features. This repo replicates the design as the origial vocos archiecture but modified to have an streaming implementation. So all the vanilla CNNs are replaced with causal CNNs and modified to work in streaming settings with dynmically adjustable chunk size (in multiples of hop size of 320ms).
What makes vocos different from other typical GAN-based vocoders is that Vocos does not model audio samples in the time domain. Instead, it generates spectral coefficients, facilitating rapid audio reconstruction through inverse Fourier transform. This cuts down the processing time significantly and is very appropriate for streaming applications that require minimal latency.
Training follows the Generative Adversarial Network (GAN) objective as original but loss functions are changed to those proposed in the descript audio codec (see repo).
Visit our demo website (coming soon) for audio samples.
Refer below for original paper and audio samples. Audio samples | Paper [abs] [pdf]
Checkpoints
Coming soon
Usage
Installation
# Clone project
git clone https://github.com/warisqr007/vocos.git
cd vocos
# [Optional] Create a conda virtual environment
conda create -n <env_name> python=3.10
conda activate <env_name>
# [Optional] Use mamba instead of conda to speed up
conda install mamba -n base -c conda-forge
# Install dependencies
pip install -r requirements.txt
Data Preprocessing
We follow the same data-processing stage as here. Please follow directions in the specified repo.
Run
Fit
python src/main.py fit -c configs/data/resynthesis.yaml -c configs/model/vocosvocoder.yaml --trainer.logger.name debug
Resume
python src/main.py fit -c configs/data/resynthesis.yaml -c configs/model/vocosvocoder.yaml --ckpt_path <ckpt_path> --trainer.logger.id exp1_id