milo-asr / README.md

merge2

d38720b 7 months ago

11.5 kB

license: openrail
language:
  - da
base_model: Qwen/Qwen3-ASR-1.7B
tags:
  - automatic-speech-recognition
  - danish
  - qwen
  - asr
  - speech-to-text
  - coral
  - streaming
datasets:
  - alexandrainst/coral
  - mozilla-foundation/common_voice_17_0
library_name: transformers
pipeline_tag: automatic-speech-recognition
metrics:
  - wer
  - cer
model-index:
  - name: hvisketiske-v2
    results:
      - task:
          type: automatic-speech-recognition
          name: Speech Recognition
        dataset:
          type: alexandrainst/coral
          name: CoRal v2 Test
          split: test
        metrics:
          - type: wer
            value: 18.47
            name: WER
          - type: cer
            value: 7.86
            name: CER

hvisketiske-v2: Danish ASR Model

hvisketiske-v2 is a state-of-the-art Danish automatic speech recognition (ASR) model based on Qwen3-ASR-1.7B, finetuned on the CoRal v2 dataset for improved Danish transcription accuracy.

Key Highlights

Feature	Value
WER on CoRal v2	18.47% (14% better than Whisper v3)
CER on CoRal v2	7.86% (11% better than Whisper v3)
Real-Time Factor	0.086 (45% faster than Whisper v3)
Model Size	~1.7B parameters

Inherited Features from Qwen3-ASR

Streaming/Real-time transcription via vLLM backend
Singing detection - can transcribe singing voice and songs with BGM
Word-level timestamps via forced alignment
30+ language support (Danish optimized)
Long audio support - up to 20 minutes per request

Performance Comparison

CoRal v2 Test Set (9,123 samples, 17.3 hours)

Model	WER	CER	RTF	Throughput	Parameters
hvisketiske-v2	18.47%	7.86%	0.086	1.71 samples/s	~1.7B
hviske-v3 (Whisper Large v3)	21.47%	8.79%	0.156	0.94 samples/s	~2B

Improvements over Whisper Large v3:

14% reduction in Word Error Rate
11% reduction in Character Error Rate
45% faster inference speed
15% fewer parameters

Comparison Plots

Quick Start

Installation

pip install qwen-asr transformers torch

Basic Usage

from qwen_asr import Qwen3ASRModel

# Load the model
model = Qwen3ASRModel.from_pretrained(
    "pluttodk/hvisketiske-v2",
    dtype="bfloat16",
    device_map="cuda:0",
)

# Transcribe audio file
results = model.transcribe(
    audio="path/to/danish_audio.wav",
    language="Danish",
)

print(results[0].text)

Advanced Usage

Batch Transcription (Fast Processing)

Process multiple audio files efficiently in a single call:

from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "pluttodk/hvisketiske-v2",
    dtype="bfloat16",
    device_map="cuda:0",
    max_inference_batch_size=16,  # Process up to 16 files at once
)

# Batch transcribe multiple files
audio_files = ["audio1.wav", "audio2.wav", "audio3.wav"]
results = model.transcribe(
    audio=audio_files,
    language="Danish",
)

for i, result in enumerate(results):
    print(f"File {i+1}: {result.text}")

Transcription with Timestamps

Get word-level timestamps using the forced aligner:

from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "pluttodk/hvisketiske-v2",
    forced_aligner="Qwen/Qwen3-ForcedAligner-0.6B",
    dtype="bfloat16",
    device_map="cuda:0",
)

results = model.transcribe(
    audio="path/to/audio.wav",
    language="Danish",
    return_time_stamps=True,
)

# Access word-level timestamps
for item in results[0].time_stamps.items:
    print(f"{item.start_time:.2f}s - {item.end_time:.2f}s: {item.text}")

Streaming/Real-time Transcription (vLLM Backend)

For real-time streaming transcription, use the vLLM backend:

from qwen_asr import Qwen3ASRModel

# Initialize with vLLM backend for streaming
model = Qwen3ASRModel.LLM(
    model="pluttodk/hvisketiske-v2",
    gpu_memory_utilization=0.8,
)

# Initialize streaming state
state = model.init_streaming_state(
    language="Danish",
    chunk_size_sec=2.0,  # Process audio in 2-second chunks
)

# Simulate streaming audio (16kHz mono float32)
import numpy as np

def audio_stream():
    """Replace with actual audio stream from microphone."""
    for chunk in audio_chunks:
        yield np.array(chunk, dtype=np.float32)

# Process streaming audio
for audio_chunk in audio_stream():
    state = model.streaming_transcribe(audio_chunk, state)
    print(f"Current transcription: {state.text}")

# Finalize stream
state = model.finish_streaming_transcribe(state)
print(f"Final transcription: {state.text}")

Using with Transformers Directly

For more control, use the model directly with transformers:

from transformers import AutoModel, AutoProcessor
import torch
import librosa

# Load model and processor
model = AutoModel.from_pretrained(
    "pluttodk/hvisketiske-v2",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",
)
processor = AutoProcessor.from_pretrained(
    "pluttodk/hvisketiske-v2",
    trust_remote_code=True,
)

# Load and preprocess audio
audio, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)

# Build input using chat template
messages = [
    {"role": "system", "content": ""},
    {"role": "user", "content": [{"type": "audio", "audio": audio}]},
]

text = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False
)
text = text + "language Danish<asr_text>"

# Process and generate
inputs = processor(text=[text], audio=[audio], return_tensors="pt", padding=True)
inputs = inputs.to(model.device).to(model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=512)
transcription = processor.batch_decode(
    output_ids[:, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)[0]

print(transcription)

Singing Detection & Multi-Audio Support

The model inherits Qwen3-ASR's ability to handle singing and background music:

from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "pluttodk/hvisketiske-v2",
    dtype="bfloat16",
    device_map="cuda:0",
)

# Transcribe audio with singing or background music
results = model.transcribe(
    audio="path/to/song.wav",
    language="Danish",  # or None for auto-detection
)

print(results[0].text)

Model Details

Model Description

hvisketiske-v2 is a Danish-specialized automatic speech recognition model created by finetuning Qwen3-ASR-1.7B on the CoRal v2 dataset. The model achieves state-of-the-art performance on Danish speech recognition while maintaining fast inference speeds.

Developed by: Mathias Oliver Valdbjørn Rønnelund
Model type: Encoder-decoder speech recognition model
Language: Danish (primary), with inherited multilingual capabilities
License: Apache 2.0
Finetuned from: Qwen/Qwen3-ASR-1.7B

Architecture

The model inherits the Qwen3-ASR architecture:

Component	Specification
Audio Encoder	24-layer transformer (1024 hidden dim, 16 attention heads)
Text Decoder	28-layer transformer (2048 hidden dim, 16 attention heads)
Total Parameters	~1.7 billion
Precision	bfloat16
Audio Input	16kHz mono WAV

Training Details

Training Data

The model was finetuned on the CoRal v2 dataset, a comprehensive Danish speech corpus containing:

Diverse Danish speakers across demographics
Various recording conditions and audio qualities
Natural conversational speech
Read-aloud speech

Training Procedure

Training Approach: Supervised Fine-Tuning (SFT) with chat template formatting

Preprocessing:

Audio resampled to 16kHz mono
Chat template applied with system prompt, audio input, and target transcription
Prefix masking to train only on transcription tokens

Training Hyperparameters:

Parameter	Value
Base model	Qwen/Qwen3-ASR-1.7B
Learning rate	2e-5
Batch size (per device)	8
Gradient accumulation steps	4
Effective batch size	32
Epochs	3
Warmup ratio	0.1
Weight decay	0.01
Max gradient norm	1.0
Precision	bfloat16
Optimizer	AdamW
LR scheduler	Linear decay
Total training steps	23,448

Hardware: Training performed on NVIDIA GPUs (~25GB GPU memory per device)

Evaluation

Test Data

Evaluated on the CoRal v2 test split:

9,123 samples
17.3 hours of audio
Diverse Danish speakers and recording conditions

Metrics

Metric	Description
WER	Word Error Rate - percentage of words incorrectly transcribed (lower is better)
CER	Character Error Rate - percentage of characters incorrectly transcribed (lower is better)
RTF	Real-Time Factor - ratio of processing time to audio duration (< 1.0 = faster than real-time)

Results Summary

Model	WER	CER	RTF	Throughput
hvisketiske-v2	18.47%	7.86%	0.086	1.71 samples/sec
hviske-v3 (Whisper v3)	21.47%	8.79%	0.156	0.94 samples/sec

Limitations

Language: Optimized for Danish; other languages may have degraded performance compared to base Qwen3-ASR
Audio quality: Best results with clear speech; noisy environments may affect accuracy
Domain: Trained on CoRal v2 which is primarily conversational/read-aloud speech; specialized domains (medical, legal, technical) may have higher error rates
Streaming: Real-time streaming requires vLLM backend installation

Intended Use

Primary Use Cases

Danish speech-to-text transcription
Subtitle generation for Danish content
Voice assistant backends
Meeting transcription
Accessibility applications

Out-of-Scope Use

Non-Danish languages (use base Qwen3-ASR instead)
Real-time speaker diarization (not supported)
Emotion/sentiment detection from speech

Citation

If you use this model, please cite:

@misc{hvisketiske-v2,
  author = {Rønnelund, Mathias Oliver Valdbjørn},
  title = {hvisketiske-v2: Danish ASR Model based on Qwen3-ASR},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/pluttodk/hvisketiske-v2}
}

Also consider citing the base model and dataset:

@article{qwen3asr,
  title={Qwen3-ASR Technical Report},
  author={Qwen Team},
  journal={arXiv preprint arXiv:2601.21337},
  year={2025}
}

@dataset{coral,
  title={CoRal: A Danish Speech Corpus},
  author={Alexandra Institute},
  year={2024},
  url={https://huggingface.co/datasets/alexandrainst/coral}
}

Acknowledgements

Qwen Team for the excellent Qwen3-ASR base model
Alexandra Institute for the CoRal v2 Danish speech corpus