README.md

---
license: openrail
language:
  - da
base_model: Qwen/Qwen3-ASR-1.7B
tags:
  - automatic-speech-recognition
  - danish
  - qwen
  - asr
  - speech-to-text
  - coral
  - streaming
datasets:
  - alexandrainst/coral
library_name: transformers
pipeline_tag: automatic-speech-recognition
metrics:
  - wer
  - cer
model-index:
  - name: milo-asr
    results:
      - task:
          type: automatic-speech-recognition
          name: Speech Recognition
        dataset:
          type: alexandrainst/coral
          name: CoRal v2 Test
          split: test
        metrics:
          - type: wer
            value: 18.47
            name: WER
          - type: cer
            value: 7.86
            name: CER
---

# Milo-ASR: Dansk ASR Model

**Milo-ASR** er en "state of the art" Dansk automatic speech recognition (ASR) model baseret på [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B), finetuned på  [CoRal v2 dataset](https://huggingface.co/datasets/alexandrainst/coral) for at gøre den bedre til at forstå dansk.

## Key Highlights

| Feature | Value |
|---------|-------|
| **WER on CoRal v2** | 18.47% (14% better than Whisper v3) |
| **CER on CoRal v2** | 7.86% (11% better than Whisper v3) |
| **Real-Time Factor** | 0.086 (45% faster than Whisper v3) |
| **Model Size** | ~1.7B parameters |

### Fordele nedarvet fra Qwen3-ASR

- **Streaming/Real-time transcription** via vLLM backend
- **Sang detection** - Til at kunne transskribere teksten fra lyde med baggrundsmusik (find ud af hvad Rasmus seebach synger 🤣)
- **Word-level timestamps** via forced alignment
- **30+ language support** (Danish optimized)
- **20 minutter pr. request** - Kan kører 20 minutter igennem pr. kald

---

## Performance Comparison

### CoRal v2 Test Set (9,123 samples, 17.3 hours)

| Model | WER | CER | RTF | Throughput | Parameters |
|-------|-----|-----|-----|------------|------------|
| **Milo-ASR** | **18.47%** | **7.86%** | **0.086** | 1.71 samples/s | ~1.7B |
| hviske-v3 (Whisper Large v3) | 21.47% | 8.79% | 0.156 | 0.94 samples/s | ~2B |

**Improvements over Whisper Large v3:**
- **14% reduction** in Word Error Rate
- **11% reduction** in Character Error Rate
- **45% faster** inference speed
- **15% fewer** parameters

### Comparison Plots

![WER Comparison](plots/wer_comparison.png)
![Speed Comparison](plots/rtf_comparison.png)
![Accuracy vs Speed](plots/accuracy_vs_speed.png)

---

## Quick Start

### Installation

```bash
pip install qwen-asr transformers torch
```

### Basic Usage

```python
from qwen_asr import Qwen3ASRModel

# Load the model
model = Qwen3ASRModel.from_pretrained(
    "pluttodk/Milo-ASR",
    dtype="bfloat16",
    device_map="cuda:0",
)

# Transcribe audio file
results = model.transcribe(
    audio="path/to/danish_audio.wav",
    language="Danish",
)

print(results[0].text)
```

---

## Advanced Usage

### Batch Transcription (Fast Processing)

Process multiple audio files efficiently in a single call:

```python
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "pluttodk/Milo-ASR",
    dtype="bfloat16",
    device_map="cuda:0",
    max_inference_batch_size=16,  # Process up to 16 files at once
)

# Batch transcribe multiple files
audio_files = ["audio1.wav", "audio2.wav", "audio3.wav"]
results = model.transcribe(
    audio=audio_files,
    language="Danish",
)

for i, result in enumerate(results):
    print(f"File {i+1}: {result.text}")
```

### Transcription with Timestamps

Get word-level timestamps using the forced aligner:

```python
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "pluttodk/Milo-ASR",
    forced_aligner="Qwen/Qwen3-ForcedAligner-0.6B",
    dtype="bfloat16",
    device_map="cuda:0",
)

results = model.transcribe(
    audio="path/to/audio.wav",
    language="Danish",
    return_time_stamps=True,
)

# Access word-level timestamps
for item in results[0].time_stamps.items:
    print(f"{item.start_time:.2f}s - {item.end_time:.2f}s: {item.text}")
```

### Streaming/Real-time Transcription (vLLM Backend)

For real-time streaming transcription, use the vLLM backend:

```python
from qwen_asr import Qwen3ASRModel

# Initialize with vLLM backend for streaming
model = Qwen3ASRModel.LLM(
    model="pluttodk/Milo-ASR",
    gpu_memory_utilization=0.8,
)

# Initialize streaming state
state = model.init_streaming_state(
    language="Danish",
    chunk_size_sec=2.0,  # Process audio in 2-second chunks
)

# Simulate streaming audio (16kHz mono float32)
import numpy as np

def audio_stream():
    """Replace with actual audio stream from microphone."""
    for chunk in audio_chunks:
        yield np.array(chunk, dtype=np.float32)

# Process streaming audio
for audio_chunk in audio_stream():
    state = model.streaming_transcribe(audio_chunk, state)
    print(f"Current transcription: {state.text}")

# Finalize stream
state = model.finish_streaming_transcribe(state)
print(f"Final transcription: {state.text}")
```

### Using with Transformers Directly

For more control, use the model directly with transformers:

```python
from transformers import AutoModel, AutoProcessor
import torch
import librosa

# Load model and processor
model = AutoModel.from_pretrained(
    "pluttodk/Milo-ASR",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",
)
processor = AutoProcessor.from_pretrained(
    "pluttodk/Milo-ASR",
    trust_remote_code=True,
)

# Load and preprocess audio
audio, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)

# Build input using chat template
messages = [
    {"role": "system", "content": ""},
    {"role": "user", "content": [{"type": "audio", "audio": audio}]},
]

text = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False
)
text = text + "language Danish<asr_text>"

# Process and generate
inputs = processor(text=[text], audio=[audio], return_tensors="pt", padding=True)
inputs = inputs.to(model.device).to(model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=512)
transcription = processor.batch_decode(
    output_ids[:, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)[0]

print(transcription)
```

### Singing Detection & Multi-Audio Support

The model inherits Qwen3-ASR's ability to handle singing and background music:

```python
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "pluttodk/Milo-ASR",
    dtype="bfloat16",
    device_map="cuda:0",
)

# Transcribe audio with singing or background music
results = model.transcribe(
    audio="path/to/song.wav",
    language="Danish",  # or None for auto-detection
)

print(results[0].text)
```

---

## Model Details

### Model Description

Milo-ASR is a Danish-specialized automatic speech recognition model created by finetuning [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) on the [CoRal v2 dataset](https://huggingface.co/datasets/alexandrainst/coral). The model achieves state-of-the-art performance on Danish speech recognition while maintaining fast inference speeds.

- **Developed by:** Mathias Oliver Valdbjørn Rønnelund
- **Model type:** Encoder-decoder speech recognition model
- **Language:** Danish (primary), with inherited multilingual capabilities
- **License:** Apache 2.0
- **Finetuned from:** [Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)

### Architecture

The model inherits the Qwen3-ASR architecture:

| Component | Specification |
|-----------|--------------|
| Audio Encoder | 24-layer transformer (1024 hidden dim, 16 attention heads) |
| Text Decoder | 28-layer transformer (2048 hidden dim, 16 attention heads) |
| Total Parameters | ~1.7 billion |
| Precision | bfloat16 |
| Audio Input | 16kHz mono WAV |

---

## Training Details

### Training Data

The model was finetuned on the [CoRal v2 dataset](https://huggingface.co/datasets/alexandrainst/coral), a comprehensive Danish speech corpus containing:
- Diverse Danish speakers across demographics
- Various recording conditions and audio qualities
- Natural conversational speech
- Read-aloud speech

### Training Procedure

**Training Approach:** Supervised Fine-Tuning (SFT) with chat template formatting

**Preprocessing:**
- Audio resampled to 16kHz mono
- Chat template applied with system prompt, audio input, and target transcription
- Prefix masking to train only on transcription tokens

**Training Hyperparameters:**

| Parameter | Value |
|-----------|-------|
| Base model | Qwen/Qwen3-ASR-1.7B |
| Learning rate | 2e-5 |
| Batch size (per device) | 8 |
| Gradient accumulation steps | 4 |
| Effective batch size | 32 |
| Epochs | 3 |
| Warmup ratio | 0.1 |
| Weight decay | 0.01 |
| Max gradient norm | 1.0 |
| Precision | bfloat16 |
| Optimizer | AdamW |
| LR scheduler | Linear decay |
| Total training steps | 23,448 |

**Hardware:** Training performed on NVIDIA GPUs (~25GB GPU memory per device)

---

## Evaluation

### Test Data

Evaluated on the CoRal v2 test split:
- **9,123 samples**
- **17.3 hours** of audio
- Diverse Danish speakers and recording conditions

### Metrics

| Metric | Description |
|--------|-------------|
| **WER** | Word Error Rate - percentage of words incorrectly transcribed (lower is better) |
| **CER** | Character Error Rate - percentage of characters incorrectly transcribed (lower is better) |
| **RTF** | Real-Time Factor - ratio of processing time to audio duration (< 1.0 = faster than real-time) |

### Results Summary

| Model | WER | CER | RTF | Throughput |
|-------|-----|-----|-----|------------|
| **Milo-ASR** | **18.47%** | **7.86%** | **0.086** | 1.71 samples/sec |
| hviske-v3 (Whisper v3) | 21.47% | 8.79% | 0.156 | 0.94 samples/sec |

---

## Citation

If you use this model, please cite:

```bibtex
@misc{Milo-ASR,
  author = {Rønnelund, Mathias Oliver Valdbjørn},
  title = {Milo-ASR: Danish ASR Model based on Qwen3-ASR},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/pluttodk/Milo-ASR}
}
```

Also consider citing the base model and dataset:

```bibtex
@article{qwen3asr,
  title={Qwen3-ASR Technical Report},
  author={Qwen Team},
  journal={arXiv preprint arXiv:2601.21337},
  year={2025}
}

@dataset{coral,
  title={CoRal: A Danish Speech Corpus},
  author={Alexandra Institute},
  year={2024},
  url={https://huggingface.co/datasets/alexandrainst/coral}
}
```

---

## Acknowledgements

- [Qwen Team](https://github.com/QwenLM) for Qwen3-ASR base model
- [Alexandra Institute](https://alexandra.dk/) for CoRal v2 lyd corpus