Initial release: parity-verified ONNX export of htdemucs_ft vocals specialist

README.md

@@ -0,0 +1,243 @@
+---
+language: en
+license: mit
+library_name: onnxruntime
+pipeline_tag: audio-to-audio
+tags:
+  - onnx
+  - onnxruntime
+  - stem-separation
+  - source-separation
+  - demucs
+  - htdemucs
+  - music
+  - audio-to-audio
+  - mobile
+  - ios
+  - android
+  - coreml
+  - directml
+  - production-ready
+  - vocal-extraction
+  - vocal-isolation
+  - vocal-remover
+  - karaoke
+  - acapella
+datasets:
+  - StemSplitio/stem-separation-benchmark-2026
+inference: false
+---
+
+# HT-Demucs FT — Vocals Specialist, ONNX
+
+**The #1 open-source vocal separator on MUSDB18-HQ**, exported to ONNX. No PyTorch required at inference. Runs on CPU / CoreML / CUDA / DirectML.
+
+This repo packages sub-model 3 of the
+[`htdemucs_ft`](https://github.com/facebookresearch/demucs) 4-bag ensemble
+as a single 316 MB `.onnx` file plus a ~150-line numpy reference inference
+script. Verified to be **numerically equivalent** to the original PyTorch
+model.
+
+> Want all 4 stems in one drop-in package? Use the full bag repo:
+> [`StemSplitio/htdemucs-ft-onnx`](https://huggingface.co/StemSplitio/htdemucs-ft-onnx).
+
+---
+
+## TL;DR
+
+```bash
+pip install onnxruntime numpy soundfile
+python infer.py your-song.mp3 ./out/
+# writes ./out/vocals.wav at 44.1 kHz stereo
+```
+
+That's it. No PyTorch, no CUDA setup, no GPU server.
+
+---
+
+## Quality
+
+| Metric (MUSDB18-HQ test, 50 songs) | Value | Source |
+|---|---|---|
+| Median vocals SDR | **9.19 dB** | [StemSplitio/stem-separation-benchmark-2026](https://huggingface.co/datasets/StemSplitio/stem-separation-benchmark-2026) |
+| Rank among open-source separators on vocals | **#1** (the highest open-source vocal SDR on MUSDB18-HQ) | same |
+| ONNX vs PyTorch max abs diff | **< 1e-3** | verified during export (see [Day 1 spike report](https://huggingface.co/StemSplitio/htdemucs-ft-drums-onnx#how-it-was-built)) |
+
+---
+
+## Performance
+
+| Runtime | Hardware | Per 7.8-s segment | Per 3-min song |
+|---|---|---:|---:|
+| **onnxruntime CPU EP** | Apple M4 Pro | **~1.6 s** | **~22 s** |
+| PyTorch CPU | Apple M4 Pro | ~2.1 s | ~29 s |
+| onnxruntime CUDA EP | NVIDIA L4 | ~0.4 s | ~5 s *(extrapolated)* |
+| onnxruntime DirectML EP | RTX 4090 | ~0.2 s | ~2 s *(extrapolated)* |
+
+**Real-time factor on M4 Pro CPU: 0.20.** Roughly 1.31× faster than
+PyTorch CPU on the same hardware.
+
+---
+
+## Common use cases
+
+- **Karaoke maker** — extract clean instrumental + acapella in one pass (pair with the `other` ONNX)
+- **Acapella extraction** — harvest isolated vocals for sampling, remixing, vocal-coach feedback
+- **Vocal removal** — build a vocal-remover app on iOS / Android / web without a GPU server
+- **Speech-from-music** — isolate spoken-word from background music for transcription
+
+---
+
+## Quick start
+
+### Python — minimal
+
+```python
+import infer
+vocals = infer.separate_vocals("your-song.mp3")
+# vocals: numpy array (2, samples) at 44.1 kHz
+```
+
+### Python — full control
+
+```python
+import soundfile as sf
+import infer
+
+# Optional execution providers — CPU is the default and most portable.
+# Swap to "coreml" on macOS, "cuda" on NVIDIA, "dml" on Windows DX12.
+audio, sr = sf.read("your-song.mp3", dtype="float32", always_2d=True)
+stems = infer.separate(audio.T, sr, providers=["CPUExecutionProvider"])
+sf.write("vocals.wav", stems[infer.SOURCES.index("vocals")].T, sr)
+```
+
+### CLI
+
+```bash
+python infer.py your-song.mp3 ./out/
+python infer.py your-song.mp3 ./out/ --providers cuda    # NVIDIA
+python infer.py your-song.mp3 ./out/ --providers coreml  # macOS
+python infer.py your-song.mp3 ./out/ --providers dml     # Windows
+```
+
+### Mobile (iOS / Swift)
+
+```swift
+import onnxruntime_objc
+
+let env = try ORTEnv(loggingLevel: .warning)
+let opts = try ORTSessionOptions()
+try opts.appendCoreMLExecutionProvider(with: ORTCoreMLExecutionProviderOptions())
+let session = try ORTSession(env: env,
+                              modelPath: Bundle.main.path(forResource: "htdemucs_ft_vocals", ofType: "onnx")!,
+                              sessionOptions: opts)
+// audio: 1 × 2 × 343980 Float32 buffer, then session.run(...).
+```
+
+### Mobile (Android / Kotlin)
+
+```kotlin
+import ai.onnxruntime.OrtEnvironment
+import ai.onnxruntime.OrtSession
+
+val env = OrtEnvironment.getEnvironment()
+val opts = OrtSession.SessionOptions().apply { addNnapi() }
+val session = env.createSession(modelPath, opts)
+```
+
+### Web (onnxruntime-web)
+
+```js
+import * as ort from "onnxruntime-web";
+const session = await ort.InferenceSession.create("htdemucs_ft_vocals.onnx", {
+  executionProviders: ["wasm"],
+  graphOptimizationLevel: "all",
+});
+const tensor = new ort.Tensor("float32", audioBuffer, [1, 2, 343980]);
+const out = await session.run({ mix: tensor });
+// out.stems.data is a Float32Array (1, 4, 2, 343980); use row 3 for vocals.
+```
+
+---
+
+## Input / output spec
+
+| Tensor | Name | Shape | Dtype | Notes |
+|---|---|---|---|---|
+| Input | `mix` | `(1, 2, 343980)` | float32 | Stereo audio, 44.1 kHz, 7.8 s segment. Values in [-1, 1]. |
+| Output | `stems` | `(1, 4, 2, 343980)` | float32 | `[drums, bass, other, vocals]` order. **Use only row 3 (`vocals`)** — the other 3 rows are weakly-predicted by-products of the vocals specialist. |
+
+For longer audio, chunk with overlap-add — see `infer.py::separate` for a
+working ~60-line implementation.
+
+---
+
+## Related repos
+
+Sibling stem-specialist ONNX repos from the same export:
+
+| Repo | Stem | Use when |
+|---|---|---|
+| [`htdemucs-ft-drums-onnx`](https://huggingface.co/StemSplitio/htdemucs-ft-drums-onnx) | drums | Drum extraction, beat transcription |
+| [`htdemucs-ft-bass-onnx`](https://huggingface.co/StemSplitio/htdemucs-ft-bass-onnx) | bass | Bassline transcription, mix rebalancing |
+| [`htdemucs-ft-other-onnx`](https://huggingface.co/StemSplitio/htdemucs-ft-other-onnx) | other | Karaoke instrumentals, sample-flipping |
+| [`htdemucs-ft-vocals-onnx`](https://huggingface.co/StemSplitio/htdemucs-ft-vocals-onnx) | vocals | **#1 open-source vocal SDR** — karaoke, acapella, vocal removal |
+| [`htdemucs-ft-onnx`](https://huggingface.co/StemSplitio/htdemucs-ft-onnx) | all 4 | Full 4-stem separation in one repo |
+
+PyTorch versions for HF Inference Endpoints:
+[`htdemucs-ft-pytorch`](https://huggingface.co/StemSplitio/htdemucs-ft-pytorch),
+[`htdemucs-ft-vocals-pytorch`](https://huggingface.co/StemSplitio/htdemucs-ft-vocals-pytorch).
+
+Full benchmark across every popular open-source separator:
+[StemSplitio/stem-separation-benchmark-2026](https://huggingface.co/datasets/StemSplitio/stem-separation-benchmark-2026).
+
+---
+
+## Skip the infrastructure — use the StemSplit API
+
+Don't want to ship a 316 MB model in your app, manage a GPU pool, or write
+overlap-add chunking? Use the **[StemSplit API](https://stemsplit.io/developers)**
+instead — same model under the hood, hosted for you, with credits and a
+dashboard.
+
+- 🌐 [stemsplit.io](https://stemsplit.io)
+- 📘 [Developer docs](https://stemsplit.io/developers/docs)
+- 🔌 [API reference](https://stemsplit.io/developers/reference)
+- 📚 [Guides & recipes](https://stemsplit.io/developers/guides)
+
+Or use the no-code tools that ship the same model family:
+
+- 🎧 [Vocal Remover](https://stemsplit.io/vocal-remover)
+- 🎧 [Karaoke Maker](https://stemsplit.io/karaoke-maker)
+- 🎧 [Acapella Maker](https://stemsplit.io/acapella-maker)
+- 🎧 [YouTube Stem Splitter](https://stemsplit.io/youtube-stem-splitter)
+
+---
+
+## Files in this repo
+
+| File | Size | Purpose |
+|---|---:|---|
+| `htdemucs_ft_vocals.onnx` | 316 MB | The exported model. Opset 17. Passes `onnx.checker`. |
+| `infer.py` | ~6 KB | Pure numpy + onnxruntime reference. No torch. |
+| `requirements.txt` | <1 KB | `onnxruntime`, `numpy`, `soundfile`. |
+| `README.md` | this file | |
+
+---
+
+## License & attribution
+
+This repo is **MIT-licensed**, matching the original HT-Demucs.
+
+```bibtex
+@inproceedings{rouard2023hybrid,
+  title     = {Hybrid Transformers for Music Source Separation},
+  author    = {Rouard, Simon and Massa, Francisco and D{\'e}fossez, Alexandre},
+  booktitle = {ICASSP},
+  year      = {2023}
+}
+```
+
+- Original PyTorch model: [`facebookresearch/demucs`](https://github.com/facebookresearch/demucs)
+- ONNX export, parity verification, and packaging by [StemSplit](https://stemsplit.io)
+- Search keywords: vocal remover onnx, karaoke maker, acapella extractor, htdemucs vocals onnx, vocal separation ios

infer.py

@@ -0,0 +1,150 @@
+"""
+Pure numpy + onnxruntime reference implementation for the HT-Demucs FT
+vocals specialist. NO TORCH at inference.
+
+Usage:
+    python infer.py input.mp3 out_dir/
+    # writes out_dir/vocals.wav
+
+Or as a library:
+    import infer
+    vocals = infer.separate_vocals("song.mp3")
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+import onnxruntime as ort
+import soundfile as sf
+
+SAMPLE_RATE = 44100
+SEGMENT_S = 7.8
+N_SAMPLES = int(SEGMENT_S * SAMPLE_RATE)
+N_CHANNELS = 2
+SOURCES = ["drums", "bass", "other", "vocals"]
+SPECIALIST_STEM = "vocals"
+DEFAULT_ONNX = Path(__file__).resolve().parent / "htdemucs_ft_vocals.onnx"
+
+
+def _make_transition_window(segment: int, overlap_frac: float = 0.25) -> np.ndarray:
+    transition = int(segment * overlap_frac)
+    window = np.ones(segment, dtype=np.float32)
+    fade = np.linspace(0, 1, transition, dtype=np.float32)
+    window[:transition] = fade
+    window[-transition:] = fade[::-1]
+    return window
+
+
+def separate(mix: np.ndarray, sample_rate: int,
+             onnx_path: Path = DEFAULT_ONNX,
+             providers: list[str] | None = None,
+             verbose: bool = True) -> np.ndarray:
+    """Run chunked overlap-add separation on a full-length mix.
+    Returns: (n_sources, channels, samples). Only the row at
+    SOURCES.index(SPECIALIST_STEM) is meaningfully predicted.
+    """
+    if sample_rate != SAMPLE_RATE:
+        raise ValueError(f"Bound to {SAMPLE_RATE} Hz; got {sample_rate}.")
+    if mix.ndim != 2 or mix.shape[0] != N_CHANNELS:
+        raise ValueError(f"Expected (2, samples) input, got {mix.shape}")
+
+    if providers is None:
+        providers = ["CPUExecutionProvider"]
+    sess = ort.InferenceSession(str(onnx_path), providers=providers)
+
+    total_len = mix.shape[1]
+    overlap = N_SAMPLES // 4
+    stride = N_SAMPLES - overlap
+    n_chunks = max(1, (total_len + stride - 1) // stride)
+
+    if verbose:
+        print(f"  input:    {total_len:,} samples ({total_len / sample_rate:.1f}s)")
+        print(f"  segment:  {N_SAMPLES:,} samples ({SEGMENT_S}s)")
+        print(f"  chunks:   {n_chunks}, provider {sess.get_providers()[0]}")
+
+    window = _make_transition_window(N_SAMPLES)
+    out = np.zeros((len(SOURCES), N_CHANNELS, total_len), dtype=np.float32)
+    weight = np.zeros(total_len, dtype=np.float32)
+
+    t0 = time.perf_counter()
+    for i in range(n_chunks):
+        start = i * stride
+        end = min(start + N_SAMPLES, total_len)
+        chunk = mix[:, start:end]
+        if chunk.shape[1] < N_SAMPLES:
+            chunk = np.pad(chunk, ((0, 0), (0, N_SAMPLES - chunk.shape[1])),
+                           mode="constant")
+        x = chunk[np.newaxis, ...].astype(np.float32)
+        stems = sess.run(["stems"], {"mix": x})[0][0]
+        chunk_len = end - start
+        w = window[:chunk_len]
+        out[:, :, start:end] += stems[:, :, :chunk_len] * w
+        weight[start:end] += w
+        if verbose:
+            print(f"    chunk {i+1}/{n_chunks}: "
+                  f"{time.perf_counter() - t0:.1f}s elapsed")
+
+    weight = np.maximum(weight, 1e-8)
+    out /= weight
+    if verbose:
+        rtf = (time.perf_counter() - t0) / (total_len / sample_rate)
+        print(f"  total:    {time.perf_counter() - t0:.2f}s (RTF {rtf:.2f})")
+    return out
+
+
+def separate_vocals(input_path: str, onnx_path: Path = DEFAULT_ONNX,
+                  providers: list[str] | None = None) -> np.ndarray:
+    """Convenience: load audio, separate, return only the vocals stem."""
+    audio, sr = sf.read(input_path, dtype="float32", always_2d=True)
+    audio = audio.T
+    if audio.shape[0] == 1:
+        audio = np.tile(audio, (2, 1))
+    elif audio.shape[0] > 2:
+        audio = audio[:2]
+    stems = separate(audio, sr, onnx_path=onnx_path, providers=providers)
+    return stems[SOURCES.index(SPECIALIST_STEM)]
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("input", type=Path)
+    ap.add_argument("out_dir", type=Path)
+    ap.add_argument("--onnx", type=Path, default=DEFAULT_ONNX)
+    ap.add_argument("--providers", type=str, default="cpu",
+                    choices=["cpu", "coreml", "cuda", "dml"])
+    ap.add_argument("--write-all-stems", action="store_true",
+                    help="Also write the (low-quality) by-product stems.")
+    args = ap.parse_args()
+
+    providers_map = {
+        "cpu":    ["CPUExecutionProvider"],
+        "coreml": ["CoreMLExecutionProvider", "CPUExecutionProvider"],
+        "cuda":   ["CUDAExecutionProvider", "CPUExecutionProvider"],
+        "dml":    ["DmlExecutionProvider", "CPUExecutionProvider"],
+    }
+    args.out_dir.mkdir(parents=True, exist_ok=True)
+
+    audio, sr = sf.read(str(args.input), dtype="float32", always_2d=True)
+    audio = audio.T
+    if audio.shape[0] == 1:
+        audio = np.tile(audio, (2, 1))
+    elif audio.shape[0] > 2:
+        audio = audio[:2]
+
+    stems = separate(audio, sr, onnx_path=args.onnx,
+                     providers=providers_map[args.providers])
+    if args.write_all_stems:
+        for i, src in enumerate(SOURCES):
+            sf.write(str(args.out_dir / f"{src}.wav"), stems[i].T, sr)
+    else:
+        target = stems[SOURCES.index(SPECIALIST_STEM)]
+        sf.write(str(args.out_dir / f"{SPECIALIST_STEM}.wav"), target.T, sr)
+        print(f"  wrote {args.out_dir / f'{SPECIALIST_STEM}.wav'}")
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,3 @@
+onnxruntime>=1.20
+numpy>=1.24
+soundfile>=0.12