Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,58 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
library_name: onnx
|
| 4 |
+
tags:
|
| 5 |
+
- audio
|
| 6 |
+
- source-separation
|
| 7 |
+
- stem-separation
|
| 8 |
+
- demucs
|
| 9 |
+
- htdemucs
|
| 10 |
+
- onnx
|
| 11 |
+
- onnxruntime-web
|
| 12 |
+
- webgpu
|
| 13 |
+
- karaoke
|
| 14 |
---
|
| 15 |
+
|
| 16 |
+
# htdemucs_ft — WebGPU / onnxruntime-web build
|
| 17 |
+
|
| 18 |
+
The **htdemucs_ft** 4-model fine-tuned ensemble (Meta's Demucs v4), exported to ONNX
|
| 19 |
+
so it **runs in the browser on WebGPU via `onnxruntime-web`** — no Python, no server.
|
| 20 |
+
|
| 21 |
+
Built for [loukai](https://github.com/monteslu/loukai)'s in-browser karaoke creator.
|
| 22 |
+
|
| 23 |
+
## What makes this different from other htdemucs ONNX
|
| 24 |
+
|
| 25 |
+
There are several htdemucs ONNX exports on the Hub already, but they're **CUDA/CPU
|
| 26 |
+
server exports** — they fail to load on the `onnxruntime-web` WebGPU execution
|
| 27 |
+
provider (in-graph STFT + many `ScatterND` ops the WebGPU EP can't place; verified:
|
| 28 |
+
session creation throws in `transformer_memcpy`). This build is shaped specifically
|
| 29 |
+
for the browser:
|
| 30 |
+
|
| 31 |
+
- **STFT/iSTFT pulled out of the graph** (done in JS), using the real-magnitude input
|
| 32 |
+
contract: `mix [1,2,343980]` + `mag [1,4,2048,336]` → `x [1,4,4,2048,336]` (freq
|
| 33 |
+
mask) + `xt [1,4,2,343980]` (time). Masking is applied in JS (see
|
| 34 |
+
[`demucs-web`](https://www.npmjs.com/package/demucs-web)).
|
| 35 |
+
- **fp16 weights** for speed/size — with the variance/normalization prologue pinned
|
| 36 |
+
to CPU (`forceCpuNodeNames`) because that op overflows fp16 on WebGPU and NaNs.
|
| 37 |
+
fp16 is parity-perfect vs fp32 (corr ~1.0).
|
| 38 |
+
- **Legacy `torch.onnx` export** (opset 17, no dynamo) — the dynamo path decomposes
|
| 39 |
+
ops in ways that NaN on WebGPU.
|
| 40 |
+
|
| 41 |
+
## Files
|
| 42 |
+
|
| 43 |
+
- `htdemucs_ft_{drums,bass,other,vocals}_safe16.onnx` — the 4 specialist models
|
| 44 |
+
(~84 MB each, fp16). Stem *k* is taken from model *k* (the bag's one-hot weights).
|
| 45 |
+
- `ft_cpu_nodes.json` — per-stem `forceCpuNodeNames` lists.
|
| 46 |
+
|
| 47 |
+
## Usage
|
| 48 |
+
|
| 49 |
+
Runs via [`loukai-htdemucs-ft` ensemble runner](https://github.com/monteslu/loukai)
|
| 50 |
+
(`createEnsembleSessions` / `runEnsemble`) on top of `demucs-web` for the STFT. See
|
| 51 |
+
the loukai repo for the full in-browser pipeline (Demucs + Whisper + CREPE, all
|
| 52 |
+
WebGPU).
|
| 53 |
+
|
| 54 |
+
## Credit
|
| 55 |
+
|
| 56 |
+
Models exported from [Demucs](https://github.com/facebookresearch/demucs)
|
| 57 |
+
(`htdemucs_ft`, MIT). Export approach builds on the timcsy / gianlourbano
|
| 58 |
+
demucs-web-onnx work.
|