pluttodk
/

milo-asr

@@ -37,53 +37,58 @@ model-index:
             name: CER
 ---
-# Milo-ASR: Dansk ASR Model
-**Milo-ASR** er en "state of the art" Dansk automatic speech recognition (ASR) model baseret på [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B), finetuned på  [CoRal v2 dataset](https://huggingface.co/datasets/alexandrainst/coral) for at gøre den bedre til at forstå dansk.
-## Key Highlights
-| Feature | Value |
 |---------|-------|
-| **WER on CoRal v2** | 18.47% (14% better than Whisper v3) |
-| **CER on CoRal v2** | 7.86% (11% better than Whisper v3) |
-| **Real-Time Factor** | 0.086 (45% faster than Whisper v3) |
-| **Model Size** | ~1.7B parameters |
-### Fordele nedarvet fra Qwen3-ASR
-- **Streaming/Real-time transcription** via vLLM backend
-- **Sang detection** - Til at kunne transskribere teksten fra lyde med baggrundsmusik (find ud af hvad Rasmus seebach synger 🤣)
-- **Word-level timestamps** via forced alignment
-- **30+ language support** (Danish optimized)
-- **20 minutter pr. request** - Kan kører 20 minutter igennem pr. kald
 ---
-## Performance Comparison
-### CoRal v2 Test Set (9,123 samples, 17.3 hours)
-| Model | WER | CER | RTF | Throughput | Parameters |
-|-------|-----|-----|-----|------------|------------|
-| **Milo-ASR** | **18.47%** | **7.86%** | **0.086** | 1.71 samples/s | ~1.7B |
-| hviske-v3 (Whisper Large v3) | 21.47% | 8.79% | 0.156 | 0.94 samples/s | ~2B |
-**Improvements over Whisper Large v3:**
-- **14% reduction** in Word Error Rate
-- **11% reduction** in Character Error Rate
-- **45% faster** inference speed
-- **15% fewer** parameters
-### Comparison Plots
-![WER Comparison](plots/wer_comparison.png)
-![Speed Comparison](plots/rtf_comparison.png)
-![Accuracy vs Speed](plots/accuracy_vs_speed.png)
 ---
-## Quick Start
 ### Installation
@@ -91,21 +96,21 @@ model-index:
 pip install qwen-asr transformers torch
 ```
-### Basic Usage
 ```python
 from qwen_asr import Qwen3ASRModel
-# Load the model
 model = Qwen3ASRModel.from_pretrained(
     "pluttodk/Milo-ASR",
     dtype="bfloat16",
     device_map="cuda:0",
 )
-# Transcribe audio file
 results = model.transcribe(
-    audio="path/to/danish_audio.wav",
     language="Danish",
 )
@@ -114,11 +119,11 @@ print(results[0].text)
 ---
-## Advanced Usage
-### Batch Transcription (Fast Processing)
-Process multiple audio files efficiently in a single call:
 ```python
 from qwen_asr import Qwen3ASRModel
@@ -127,23 +132,22 @@ model = Qwen3ASRModel.from_pretrained(
     "pluttodk/Milo-ASR",
     dtype="bfloat16",
     device_map="cuda:0",
-    max_inference_batch_size=16,  # Process up to 16 files at once
 )
-# Batch transcribe multiple files
-audio_files = ["audio1.wav", "audio2.wav", "audio3.wav"]
 results = model.transcribe(
-    audio=audio_files,
     language="Danish",
 )
 for i, result in enumerate(results):
-    print(f"File {i+1}: {result.text}")
 ```
-### Transcription with Timestamps
-Get word-level timestamps using the forced aligner:
 ```python
 from qwen_asr import Qwen3ASRModel
@@ -156,63 +160,56 @@ model = Qwen3ASRModel.from_pretrained(
 )
 results = model.transcribe(
-    audio="path/to/audio.wav",
     language="Danish",
     return_time_stamps=True,
 )
-# Access word-level timestamps
 for item in results[0].time_stamps.items:
     print(f"{item.start_time:.2f}s - {item.end_time:.2f}s: {item.text}")
 ```
-### Streaming/Real-time Transcription (vLLM Backend)
-For real-time streaming transcription, use the vLLM backend:
 ```python
 from qwen_asr import Qwen3ASRModel
-# Initialize with vLLM backend for streaming
 model = Qwen3ASRModel.LLM(
     model="pluttodk/Milo-ASR",
     gpu_memory_utilization=0.8,
 )
-# Initialize streaming state
 state = model.init_streaming_state(
     language="Danish",
-    chunk_size_sec=2.0,  # Process audio in 2-second chunks
 )
-# Simulate streaming audio (16kHz mono float32)
 import numpy as np
-def audio_stream():
-    """Replace with actual audio stream from microphone."""
     for chunk in audio_chunks:
         yield np.array(chunk, dtype=np.float32)
-# Process streaming audio
-for audio_chunk in audio_stream():
-    state = model.streaming_transcribe(audio_chunk, state)
-    print(f"Current transcription: {state.text}")
-# Finalize stream
 state = model.finish_streaming_transcribe(state)
-print(f"Final transcription: {state.text}")
 ```
-### Using with Transformers Directly
-For more control, use the model directly with transformers:
 ```python
 from transformers import AutoModel, AutoProcessor
 import torch
 import librosa
-# Load model and processor
 model = AutoModel.from_pretrained(
     "pluttodk/Milo-ASR",
     trust_remote_code=True,
@@ -224,10 +221,8 @@ processor = AutoProcessor.from_pretrained(
     trust_remote_code=True,
 )
-# Load and preprocess audio
-audio, sr = librosa.load("path/to/audio.wav", sr=16000, mono=True)
-# Build input using chat template
 messages = [
     {"role": "system", "content": ""},
     {"role": "user", "content": [{"type": "audio", "audio": audio}]},
@@ -240,22 +235,21 @@ text = processor.apply_chat_template(
 )
 text = text + "language Danish<asr_text>"
-# Process and generate
 inputs = processor(text=[text], audio=[audio], return_tensors="pt", padding=True)
 inputs = inputs.to(model.device).to(model.dtype)
 output_ids = model.generate(**inputs, max_new_tokens=512)
-transcription = processor.batch_decode(
     output_ids[:, inputs["input_ids"].shape[1]:],
     skip_special_tokens=True,
 )[0]
-print(transcription)
 ```
-### Singing Detection & Multi-Audio Support
-The model inherits Qwen3-ASR's ability to handle singing and background music:
 ```python
 from qwen_asr import Qwen3ASRModel
@@ -266,10 +260,9 @@ model = Qwen3ASRModel.from_pretrained(
     device_map="cuda:0",
 )
-# Transcribe audio with singing or background music
 results = model.transcribe(
-    audio="path/to/song.wav",
-    language="Danish",  # or None for auto-detection
 )
 print(results[0].text)
@@ -277,102 +270,105 @@ print(results[0].text)
 ---
-## Model Details
-### Model Description
-Milo-ASR is a Danish-specialized automatic speech recognition model created by finetuning [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) on the [CoRal v2 dataset](https://huggingface.co/datasets/alexandrainst/coral). The model achieves state-of-the-art performance on Danish speech recognition while maintaining fast inference speeds.
-- **Developed by:** Mathias Oliver Valdbjørn Rønnelund
-- **Model type:** Encoder-decoder speech recognition model
-- **Language:** Danish (primary), with inherited multilingual capabilities
-- **License:** Apache 2.0
-- **Finetuned from:** [Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
-### Architecture
-The model inherits the Qwen3-ASR architecture:
-| Component | Specification |
 |-----------|--------------|
-| Audio Encoder | 24-layer transformer (1024 hidden dim, 16 attention heads) |
-| Text Decoder | 28-layer transformer (2048 hidden dim, 16 attention heads) |
-| Total Parameters | ~1.7 billion |
-| Precision | bfloat16 |
-| Audio Input | 16kHz mono WAV |
 ---
-## Training Details
-### Training Data
-The model was finetuned on the [CoRal v2 dataset](https://huggingface.co/datasets/alexandrainst/coral), a comprehensive Danish speech corpus containing:
-- Diverse Danish speakers across demographics
-- Various recording conditions and audio qualities
-- Natural conversational speech
-- Read-aloud speech
-### Training Procedure
-**Training Approach:** Supervised Fine-Tuning (SFT) with chat template formatting
-**Preprocessing:**
-- Audio resampled to 16kHz mono
-- Chat template applied with system prompt, audio input, and target transcription
-- Prefix masking to train only on transcription tokens
-**Training Hyperparameters:**
-| Parameter | Value |
 |-----------|-------|
-| Base model | Qwen/Qwen3-ASR-1.7B |
-| Learning rate | 2e-5 |
-| Batch size (per device) | 8 |
 | Gradient accumulation steps | 4 |
-| Effective batch size | 32 |
-| Epochs | 3 |
-| Warmup ratio | 0.1 |
-| Weight decay | 0.01 |
-| Max gradient norm | 1.0 |
-| Precision | bfloat16 |
 | Optimizer | AdamW |
-| LR scheduler | Linear decay |
-| Total training steps | 23,448 |
-**Hardware:** Training performed on NVIDIA GPUs (~25GB GPU memory per device)
 ---
-## Evaluation
-### Test Data
-Evaluated on the CoRal v2 test split:
-- **9,123 samples**
-- **17.3 hours** of audio
-- Diverse Danish speakers and recording conditions
-### Metrics
-| Metric | Description |
 |--------|-------------|
-| **WER** | Word Error Rate - percentage of words incorrectly transcribed (lower is better) |
-| **CER** | Character Error Rate - percentage of characters incorrectly transcribed (lower is better) |
-| **RTF** | Real-Time Factor - ratio of processing time to audio duration (< 1.0 = faster than real-time) |
-### Results Summary
-| Model | WER | CER | RTF | Throughput |
-|-------|-----|-----|-----|------------|
-| **Milo-ASR** | **18.47%** | **7.86%** | **0.086** | 1.71 samples/sec |
-| hviske-v3 (Whisper v3) | 21.47% | 8.79% | 0.156 | 0.94 samples/sec |
 ---
-## Citation
-If you use this model, please cite:
 ```bibtex
 @misc{Milo-ASR,
@@ -384,7 +380,7 @@ If you use this model, please cite:
 }
 ```
-Also consider citing the base model and dataset:
 ```bibtex
 @article{qwen3asr,
@@ -404,7 +400,7 @@ Also consider citing the base model and dataset:
 ---
-## Acknowledgements
-- [Qwen Team](https://github.com/QwenLM) for Qwen3-ASR base model
-- [Alexandra Institute](https://alexandra.dk/) for CoRal v2 lyd corpus

             name: CER
 ---
+# Milo-ASR: Dansk talegenkendelses model
+**Milo-ASR** er en dansk ASR-model bygget oven på [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B), som er finetunet på [CoRal v2-datasættet](https://huggingface.co/datasets/alexandrainst/coral) for at blive god til dansk.
+## Kort fortalt
+| Feature | Værdi |
 |---------|-------|
+| **WER på CoRal v2** | 18,47% (14% bedre end hviske-v3-conversation) |
+| **CER på CoRal v2** | 7,86% (11% bedre end hviske-v3-conversation) |
+| **Real-Time Factor** | 0,087 (43% hurtigere end hviske-v3-conversation) |
+| **Modelstørrelse** | ~1,7B parametre |
+### Hvad Milo-ASR kan (nedarvet fra Qwen3-ASR)
+- **Streaming/realtidstransskription** via vLLM backend
+- **Sanggenkendelse** – Milo-ASR kan transskribere tale i lyd med baggrundsmusik
+- **Tidsstempler på ordniveau** via forced alignment
+- **30+ sprog** (optimeret til dansk)
+- **Op til 20 minutter pr. kald** – du behøver ikke hakke lyden op i småbidder
 ---
+## Sammenligning med andre modeller
+### CoRal v2 testsæt (9.123 eksempler, ~17,3 timer)
+| Model | WER | CER | RTF | Gennemløb | Parametre |
+|-------|-----|-----|-----|-----------|-----------|
+| **Milo-ASR** | **18,47%** | **7,86%** | **0,087** | **1,69 eks./s** | ~1,7B |
+| hviske-v2 (Whisper Large v2) | 12,74% | 4,94% | 0,154 | 0,95 eks./s | ~1,5B |
+| hviske-v3-conversation (Whisper Large v3) | 21,47% | 8,79% | 0,153 | 0,95 eks./s | ~2B |
+| Whisper Large v3 Turbo | 38,54% | 13,73% | 0,064 | 2,29 eks./s | ~0,8B |
+| Qwen3-ASR-1.7B (base) | 46,03% | 18,85% | 0,100 | 1,46 eks./s | ~1,7B |
+**Milo-ASR vs. hviske-v3-conversation (Whisper Large v3):**
+- **14% lavere** Word Error Rate (18,47% vs. 21,47%)
+- **11% lavere** Character Error Rate (7,86% vs. 8,79%)
+- **43% hurtigere** inferens (RTF: 0,087 vs. 0,153)
+- **15% færre** parametre (~1,7B vs. ~2B)
+> **Bemærk:** hviske-v2 (Whisper Large v2) klarer sig bedre end Milo-ASR på WER og CER. Til gengæld er Milo-ASR næsten dobbelt så hurtig (RTF 0,087 vs. 0,154), så hvis hastighed er vigtig for dig, er Milo-ASR det oplagte valg.
+### Sammenligningsplots
+![WER sammenligning](plots/wer_comparison.png)
+![Hastighed sammenligning](plots/rtf_comparison.png)
+![Nøjagtighed vs. hastighed](plots/accuracy_vs_speed.png)
 ---
+## Kom i gang
 ### Installation
 pip install qwen-asr transformers torch
 ```
+### Basis brug
 ```python
 from qwen_asr import Qwen3ASRModel
+# Indlæs Milo-ASR
 model = Qwen3ASRModel.from_pretrained(
     "pluttodk/Milo-ASR",
     dtype="bfloat16",
     device_map="cuda:0",
 )
+# Transskriber en dansk lydfil
 results = model.transcribe(
+    audio="sti/til/dansk_lyd.wav",
     language="Danish",
 )
 ---
+## Avanceret brug
+### Batch-transskription
+Kør flere filer på én gang:
 ```python
 from qwen_asr import Qwen3ASRModel
     "pluttodk/Milo-ASR",
     dtype="bfloat16",
     device_map="cuda:0",
+    max_inference_batch_size=16,
 )
+audio_filer = ["lyd1.wav", "lyd2.wav", "lyd3.wav"]
 results = model.transcribe(
+    audio=audio_filer,
     language="Danish",
 )
 for i, result in enumerate(results):
+    print(f"Fil {i+1}: {result.text}")
 ```
+### Transskription med tidsstempler
+Få tidsstempler på ordniveau via forced aligner:
 ```python
 from qwen_asr import Qwen3ASRModel
 )
 results = model.transcribe(
+    audio="sti/til/lyd.wav",
     language="Danish",
     return_time_stamps=True,
 )
 for item in results[0].time_stamps.items:
     print(f"{item.start_time:.2f}s - {item.end_time:.2f}s: {item.text}")
 ```
+### Streaming i realtid (vLLM backend)
+Hvis du vil have live transskription, fx fra en mikrofon:
 ```python
 from qwen_asr import Qwen3ASRModel
 model = Qwen3ASRModel.LLM(
     model="pluttodk/Milo-ASR",
     gpu_memory_utilization=0.8,
 )
 state = model.init_streaming_state(
     language="Danish",
+    chunk_size_sec=2.0,
 )
 import numpy as np
+def lyd_stream():
+    """Erstat med din faktiske lydstream fra mikrofon."""
     for chunk in audio_chunks:
         yield np.array(chunk, dtype=np.float32)
+for lyd_chunk in lyd_stream():
+    state = model.streaming_transcribe(lyd_chunk, state)
+    print(f"Løbende transskription: {state.text}")
 state = model.finish_streaming_transcribe(state)
+print(f"Endelig transskription: {state.text}")
 ```
+### Direkte brug med Transformers
+Vil du have fuld kontrol over modellen, kan du bruge Transformers direkte:
 ```python
 from transformers import AutoModel, AutoProcessor
 import torch
 import librosa
 model = AutoModel.from_pretrained(
     "pluttodk/Milo-ASR",
     trust_remote_code=True,
     trust_remote_code=True,
 )
+audio, sr = librosa.load("sti/til/lyd.wav", sr=16000, mono=True)
 messages = [
     {"role": "system", "content": ""},
     {"role": "user", "content": [{"type": "audio", "audio": audio}]},
 )
 text = text + "language Danish<asr_text>"
 inputs = processor(text=[text], audio=[audio], return_tensors="pt", padding=True)
 inputs = inputs.to(model.device).to(model.dtype)
 output_ids = model.generate(**inputs, max_new_tokens=512)
+transskription = processor.batch_decode(
     output_ids[:, inputs["input_ids"].shape[1]:],
     skip_special_tokens=True,
 )[0]
+print(transskription)
 ```
+### Sang og baggrundsmusik
+Milo-ASR kan håndtere lyd med sang eller baggrundsmusik:
 ```python
 from qwen_asr import Qwen3ASRModel
     device_map="cuda:0",
 )
 results = model.transcribe(
+    audio="sti/til/sang.wav",
+    language="Danish",  # eller None for automatisk sproggenkendelse
 )
 print(results[0].text)
 ---
+## Om modellen
+### Beskrivelse
+Milo-ASR er lavet ved at finetunet [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) på [CoRal v2-datasættet](https://huggingface.co/datasets/alexandrainst/coral). Resultatet er en model, der er skarp til dansk tale og samtidig hurtig nok til at bruges i produktionsmiljøer og realtidsapplikationer.
+- **Udviklet af:** Mathias Oliver Valdbjørn Rønnelund
+- **Modeltype:** Encoder-decoder talegenkendelses model
+- **Sprog:** Dansk (primært), med nedarvet flersproget understøttelse
+- **Licens:** OpenRAIL
+- **Finetunet fra:** [Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
+### Arkitektur
+| Komponent | Specifikation |
 |-----------|--------------|
+| Audio Encoder | 24-lags transformer (1024 hidden dim, 16 attention heads) |
+| Text Decoder | 28-lags transformer (2048 hidden dim, 16 attention heads) |
+| Parametre i alt | ~1,7 milliarder |
+| Præcision | bfloat16 |
+| Lydinput | 16kHz mono WAV |
 ---
+## Træning
+### Træningsdata
+Milo-ASR er finetunet på [CoRal v2-datasættet](https://huggingface.co/datasets/alexandrainst/coral), som er et dansk talekorpus med:
+- Mange forskellige danske talere på tværs af alder, køn og dialekter
+- Varierende optagelseskvalitet og lydmiljøer
+- Naturlig samtaletale
+- Oplæst tale
+### Træningsopsætning
+**Tilgang:** Supervised Fine-Tuning (SFT) med chat template-formatering
+**Forbehandling:**
+- Lyd resamplet til 16kHz mono
+- Chat template anvendt med systemprompt, lydinput og måltransskription
+- Prefix masking så modellen kun trænes på transskriptionstokens
+**Hyperparametre:**
+| Parameter | Værdi |
 |-----------|-------|
+| Basismodel | Qwen/Qwen3-ASR-1.7B |
+| Læringsrate | 2e-5 |
+| Batchstørrelse (per device) | 8 |
 | Gradient accumulation steps | 4 |
+| Effektiv batchstørrelse | 32 |
+| Epoker | 3 |
+| Warmup ratio | 0,1 |
+| Weight decay | 0,01 |
+| Max gradient norm | 1,0 |
+| Præcision | bfloat16 |
 | Optimizer | AdamW |
+| LR scheduler | Lineært fald |
+| Træningsskridt i alt | 23.448 |
+**Hardware:** NVIDIA GPU'er (~25GB GPU-hukommelse per device)
 ---
+## Evaluering
+### Testdata
+Milo-ASR er evalueret på CoRal v2-testsættet:
+- **9.123 eksempler**
+- **~17,3 timers** lyd
+- Bred repræsentation af danske talere og optagelsesforhold
+### Metrikker
+| Metrik | Beskrivelse |
 |--------|-------------|
+| **WER** | Word Error Rate – andelen af forkert transskriberede ord (lavere er bedre) |
+| **CER** | Character Error Rate – andelen af forkert transskriberede tegn (lavere er bedre) |
+| **RTF** | Real-Time Factor – forholdet mellem procestid og lydvarighed (under 1,0 = hurtigere end realtid) |
+### Resultater
+| Model | WER | CER | RTF | Gennemløb | Parametre |
+|-------|-----|-----|-----|-----------|-----------|
+| **Milo-ASR** | **18,47%** | **7,86%** | **0,087** | **1,69 eks./s** | ~1,7B |
+| hviske-v2 (Whisper Large v2) | 12,74% | 4,94% | 0,154 | 0,95 eks./s | ~1,5B |
+| hviske-v3-conversation (Whisper Large v3) | 21,47% | 8,79% | 0,153 | 0,95 eks./s | ~2B |
+| Whisper Large v3 Turbo | 38,54% | 13,73% | 0,064 | 2,29 eks./s | ~0,8B |
+| Qwen3-ASR-1.7B (base) | 46,03% | 18,85% | 0,100 | 1,46 eks./s | ~1,7B |
+Milo-ASR slår hviske-v3-conversation med 14% på WER og 11% på CER, og er samtidig 43% hurtigere. Sammenlignet med den utrænede Qwen3-ASR-1.7B basismodel falder WER fra 46,03% til 18,47% – et klart tegn på, at finetuning på CoRal v2 gør en stor forskel.
 ---
+## Citér modellen
+Bruger du Milo-ASR i dit projekt, må du meget gerne citere:
 ```bibtex
 @misc{Milo-ASR,
 }
 ```
+Og gerne også basismodellen og datasættet:
 ```bibtex
 @article{qwen3asr,
 ---
+## Tak til
+- [Qwen-teamet](https://github.com/QwenLM) for Qwen3-ASR basismodellen
+- [Alexandra Instituttet](https://alexandra.dk/) for CoRal v2-datasættet