surus-ai/cohere-transcribe-spanish-MegaASR

DG-WGPO Noise-Robust ASR Adapter

Fine-tuned LoRA adapter by Surus for noise-robust Spanish speech recognition using Dual-Granularity WER-Gated Policy Optimization (DG-WGPO).

Model Details

Model Description

This is a LoRA adapter on top of CohereLabs/cohere-transcribe-03-2026 that improves transcription accuracy on degraded/noisy audio by 57.5% relative while preserving clean-speech performance. The adapter is trained using DG-WGPO, a DAPO-style reinforcement learning method with dual-granularity WER-gated rewards from the Mega-ASR paper (arXiv:2605.19833).

• Developed by: Marian Basti @ SURUS
• Model type: LoRA adapter for sequence-to-sequence speech recognition
• Language(s): Spanish (es)
• License: Apache-2.0
• Base model: CohereLabs/cohere-transcribe-03-2026

Model Sources

• Paper: Mega-ASR: Towards Building End-to-End Large-Scale Multilingual ASR Systems

Uses

Direct Use

Transcribe Spanish audio that may contain background noise, reverberation, or other acoustic degradation. The adapter is specifically optimized for noisy environments where the base model's accuracy degrades.

Downstream Use

Can be used as a noise-robust ASR backbone for Spanish speech applications.

Out-of-Scope Use

• Non-Spanish languages (trained exclusively on Common Voice Spanish)
• Audio longer than 30 seconds (chunking required)
• Real-time streaming (not optimized for latency)

Bias, Risks, and Limitations

• Trained on Common Voice Spanish, which has known demographic biases (age, gender, dialect representation)
• The model was fine-tuned with online augmentation (noise, reverb, speed perturbation, etc.) but may not generalize to all degradation types
• RL training with KL regularization means the adapter stays close to the base model; extreme domain shift may not be handled

Recommendations

• Use with the companion router for best results: route clean audio to the base model, degraded audio to this adapter
• For audio with very heavy degradation (e.g., overlapping speakers, extremely low SNR), results may vary

How to Get Started with the Model

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
from peft import PeftModel
import torch

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    "CohereLabs/cohere-transcribe-03-2026",
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(model, "surus-ai/cohere-transcribe-spanish-MegaASR")
processor = AutoProcessor.from_pretrained("CohereLabs/cohere-transcribe-03-2026")

# Process audio
inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt", language="es")
with torch.no_grad():
    outputs = model.generate(
        input_features=inputs.input_features,
        attention_mask=inputs.attention_mask,
        max_new_tokens=256,
    )
transcription = processor.decode(outputs[0], skip_special_tokens=True)

Training Details

Training Data

• Dataset: Common Voice Spanish v26.0 (released 2026-06-12)
  • 332,538 training clips
  • 16,000+ dev clips (500 used for evaluation)
  • ~1.1 million mp3 audio files in total
  • All audio clips are crowd-sourced Spanish speech in mp3 format

Preprocessing Pipeline

Audio loading and resampling:

1. Load mp3 via torchaudio.load()
2. Convert stereo to mono by averaging channels
3. Resample to 16 kHz mono via torchaudio.functional.resample()

Text normalization:

1. Strip leading/trailing whitespace
2. Collapse multiple whitespace to single space
3. Lowercase all text
4. Remove characters not in [a-záéíóúñü0-9\s.,;:!?¡¿"'-]
5. Truncate transcripts to 256 tokens max (tokenizer limit)

Feature extraction (via Cohere processor, trust_remote_code=False):

1. Audio waveform (16 kHz float32 numpy array) passed to the processor
2. Processor extracts 128-bin log-Mel spectrogram features
3. Output format: (B, T, 128) time-major (NOT channel-major) — critical for Parakeet Conv2D subsampling
4. T is unconstrained — variable-length padding handled by attention masking

Collation and padding:

1. Features padded along time dimension (dim 0) to max length in batch: (B, max_T, 128)
2. Attention mask generated as (B, max_T) float tensor: 1.0 for valid frames, 0.0 for padding
3. Labels padded with -100 (ignored by CrossEntropyLoss)
4. Labels are tokenized via the processor's tokenizer with max_length=256, truncation=True, padding="max_length"

Filtering:

• Clips filtered by duration: 0.5s ≤ duration ≤ 30s
• Clips with duration = 0 (missing metadata) are included as fallback
• Empty transcripts excluded

Augmentation Pipeline (Online)

Augmentations are applied online during training (in __getitem__), providing infinite variety per epoch. No offline preprocessing required.

11 augmentation effects (7 from Mega-ASR paper + 4 extras):

#	Effect	Description
1	Additive noise	MUSAN noise mixed at target SNR (babble, environmental, music)
2	Far-field	Distance attenuation + low-pass air absorption filter
3	Obstructed	Random spectral band attenuation (simulating physical barriers)
4	Reverb	Convolution with synthetic RIRs (pyroomacoustics, 10 room types)
5	Recording coloration	Biquad filter peaks (simulating microphone frequency response)
6	Electronic distortion	Soft/hard clipping + tanh saturation
7	Transmission dropout	Frame-level silence insertion (packet loss)
8	Speed perturbation	Time-stretch via resampling
9	Volume perturbation	Random gain ±10 dB
10	Frequency masking	Zero random frequency bins
11	Time masking	Zero random time segments

Noise data sources:

• MUSAN corpus: 426 babble, 930 environmental, 660 music files
• Synthetic RIRs (pyroomacoustics): 1,999 files across 10 room types (1-24 m², various reverberation times)

Heavy augmentation intensity (used for DG-WGPO training):

Parameter	Range
SNR	0–15 dB
Speed	0.85–1.15
Reverb RT60	0.3–0.8 s
Far-field attenuation	10–25 dB
Distortion gain	0–6 dB
Dropout probability	20%
Compound chain	4 effects, prob=0.7
Per-sample application	prob=1.0 (all samples augmented)

Training Procedure

This model was trained using Dual-Granularity WER-Gated Policy Optimization (DG-WGPO), a reinforcement learning method from the Mega-ASR paper.

Why RL instead of supervised fine-tuning? The base model already achieves 5.08% clean WER. Supervised fine-tuning (A2S-SFT) with progressive acoustic adaptation was attempted 3 times and failed: it destroyed clean performance without improving degraded WER. DG-WGPO instead uses WER-gated rewards to optimize the model directly on degraded audio.

Training Hyperparameters

Parameter	Value
Method	DG-WGPO (DAPO-style RL)
Base model	CohereLabs/cohere-transcribe-03-2026
LoRA rank / alpha / dropout	8 / 16 / 0.05
Target modules	q_proj, v_proj, k_proj, o_proj, fc1, fc2 (encoder + decoder)
Modules to save	proj_out
Trainable params	22.4M (1.07% of 2.08B total)
Learning rate	1e-5
KL regularization weight (beta_kl)	0.1
Rollouts per batch (K)	16
Temperature	0.5
Top-k / Top-p	50 / 0.95
Repetition penalty	1.08
DAPO clip epsilon (low/high)	0.2 / 0.28
Reward structure	R = 0.4 * R_static + 0.6 * R_dynamic
WER gate tau	0.3
Batch size per GPU	2 (effective: 4 across 2 GPUs)
Gradient accumulation	16
Max training steps	450 (best checkpoint)
Warmup ratio	3%
Max gradient norm	1.0
Precision	bf16 mixed
Training data	Common Voice Spanish with heavy augmentation

Reward Components

• R_static = R_rep * R_wer (anti-repetition gate * WER-based reward)
• R_dynamic = WER-gated(R_fine, R_struc) (token-level refinement + sentence-level reconstruction)
• R = 0.4 * R_static + 0.6 * R_dynamic

Speeds, Sizes, Times

• Training time: ~10 hours on 2x NVIDIA RTX 3090
• Adapter size: 54 MB
• Training steps: 450 / 5195 (best checkpoint selected by degraded dev WER)
• NaN events: ~15 (handled gracefully with zero-loss replacement)

Evaluation

Testing Data

Common Voice Spanish dev set (500 samples), evaluated under both clean and heavily-augmented conditions.

Metrics

• Word Error Rate (WER) on clean audio
• Word Error Rate (WER) on heavily-augmented audio (online augmentation with noise, reverb, speed perturbation, etc.)

Results

Condition	Baseline (pretrained)	DG-WGPO (this model)	Relative Improvement
Clean audio	5.08%	5.66%	-11.4% (stable, within noise)
Heavy augmentation	22.8%	9.68%	-57.5%

The adapter achieves a 57.5% relative reduction in WER on degraded audio while maintaining clean-speech performance within 0.6 percentage points of the baseline.

Summary

The DG-WGPO fine-tuning successfully improves noise robustness without sacrificing clean transcription accuracy. The key insight was training directly on degraded audio with KL regularization against the frozen pretrained model, rather than the paper's recommended progressive A2S-SFT approach which proved destructive for this already-strong base model.

Technical Specifications

Model Architecture and Objective

• Architecture: Parakeet Conformer encoder (1.9B params, 48 layers) + Transformer decoder (153M params, 8 layers) with LoRA adapters
• Objective: Minimize WER on degraded Spanish audio via DAPO-style policy optimization with WER-gated dual-granularity rewards
• Adapter type: LoRA (Low-Rank Adaptation) with rank=8, applied to attention projections and feed-forward layers in both encoder and decoder

Compute Infrastructure

Hardware

• 2x NVIDIA RTX 3090 (24 GB each)

Software

• Python 3.10
• PyTorch 2.12.1+cu130
• Transformers 5.12.1
• PEFT 0.19.1
• Accelerate (for DDP)

Citation

BibTeX:

@misc{xie2026megaasr,
      title={Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation},
      author={Zhifei Xie and Kaiyu Pang and Haobin Zhang and Deheng Ye and Xiaobin Hu and Shuicheng Yan and Chunyan Miao},
      year={2026},
      eprint={2605.19833},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2605.19833},
}

Framework versions

• PEFT 0.19.1
• Transformers 5.12.1
• PyTorch 2.12.1+cu130