Malfaro43/Realtime-REUSE-whisper-finetune

Realtime RE-USE — Whisper Finetune

A fine-tuned version of NVIDIA Real-time RE-USE optimised for enhancing whispered speech and noisy recordings in ASR pipelines.

Usage

import torch
from Realtime_REUSE.models.streaming_generator_SEMamba_time_d1_random_layer_ahead_sep_conv import (
    SEMamba_decoder_list,
)

model = SEMamba_decoder_list.from_pretrained("{repo_id}")
model.eval()

Or via the CLI:

python -m scripts.inference_realtime_reuse \
    --input ./noisy_audio/ \
    --output ./enhanced_audio/ \
    --checkpoint {repo_id}/pytorch_model.bin

Model Architecture

Property	Value
Base model	NVIDIA Real-time RE-USE (SEMamba, 12 layers, 3.7 M params)
Architecture	Conv encoder + Mamba blocks + Conv decoders
Fine-tuning data	Whisper-format speech (normal + whispered + background noise)

Training Details

• Base checkpoint: NVIDIA Real-time RE-USE USEMamba_12x1_lr_00002_norm_05_vq_067_nfft_320_hop_160_NRIR_012_pha_0005_com_04_early_005_release_random_layer_GAN_longer_1k
• Optimiser: Optuna-tuned learning rate, early stopping
• Exit layer: 8 (configurable at inference time)

Input / Output

• Input: Mono-channel audio, 8 kHz – 48 kHz (.wav, .flac)
• Output: Enhanced mono-channel audio (.flac)

License

This model is derived from NVIDIA Real-time RE-USE, released under the NVIDIA One-Way Noncommercial License (NSCLv1).

References

• Rethinking Training Targets, Architectures and Data Quality for Universal Speech Enhancement
• One Model, Many Latencies: Universal Speech Enhancement for Diverse Real-Time Applications