Beat This! — beat & downbeat tracking (ONNX, WebGPU)

ONNX export of Beat This!, the JKU beat/downbeat tracker (checkpoint final0), packaged for the musetric packages/ai runtime on onnxruntime-web's WebGPU execution provider.

The graph covers the network only:

beat_this.onnx  spect [windows, 513, 128] → beat, downbeat [windows, 513]

Feature extraction is the host's job, and mel-filterbank.bin is what it needs to do it: the host computes the log-mel spectrogram on the GPU (STFT + this filterbank + log1p) and feeds windows straight into the graph as GPU buffers, never round-tripping features through the CPU.

Feed one 513-frame window per call. Longer tracks are covered by overlapping windows in the host (chunkSize 513, borderSize 6). Batching every window into a single call materializes a huge time-attention tensor; one window per call is why the graph takes a window axis at all. Short clips must be padded to 513 frames.

The export substitutes no algorithm. The one construct that cannot be traced — PartialFTTransformer.forward's b = len(x), which would freeze the window count into the graph — is re-expressed with x.shape, and is bit-identical to the original in torch. So the graph carries no approximation and its parity does not depend on the material.

Intended uses & limitations

Intended:

Beat and downbeat times for a music track, as a stage in an audio pipeline; tempo and meter are derived from them host-side.
GPU inference through onnxruntime-web. The wasm EP also works and gives identical results, at roughly 12× the wall clock.

Out of scope:

Use in other training frameworks — this is an inference-only export.
The madmom DBN postprocessor. This export targets the reference's minimal postprocessing, which is what Beat This! is designed around.

Limitations:

The host owns feature extraction, and the contract is exact. 22050 Hz mono; channels downmixed as the arithmetic mean, matching the reference Audio2Frames; STFT with n_fft 1024, hop 441, a periodic Hann window, center=True with reflect padding; magnitude divided by sqrt(n_fft) (torchaudio's normalized='frame_length'); projected onto mel-filterbank.bin; then log1p(1000 · x). config.json records all of it.
ffmpeg -ac 1 is the wrong downmix. It is an energy-preserving rematrix and comes out √2 louder; because features are log1p(1000 · mel), that gain is not a constant offset and will move beats. config.json says "downmix": "mean".
The host also owns the chunk layout. The graph does not know about chunkSize, borderSize or the shifted final window. Reproduce split_piece / aggregate_prediction (keep_first) or results drift at window seams.
Validate on the material fed in production. For this pipeline that is the instrumental stem.
Training-data provenance of the upstream checkpoint is not documented here.

Known defect: onnxruntime WebGPU on Adreno, and the static rewrite (2026-09-12)

Strict on-device verification (one recorded input, wasm EP of the same build as the bit-exact reference, comparator validated by negative controls) showed the WebGPU EP returns silently wrong fp32 logits for the original dynamic graph on mobile Adreno:

Adreno 660: shared-tile Transpose/Conv (a quarter of the output left as zeros) and inference BatchNormalization (~50% mismatch, nondeterministic).
Adreno 660 and 750: the 1500×1500 time-attention MatMul (13.5% mismatch at N=1500; isolated MatMuls with N≤512 are bit-exact). Desktop NVIDIA is bit-exact on the same input.

beat_this.onnx now ships the static 513-frame rewrite: broken Transpose families become Reshape -> Gather(const int32 idx) -> Reshape, every Conv becomes Pad + im2col Gather + MatMul, and the stem BatchNorm is folded to Mul+Add. Verified on 9RT WebGPU against wasm of the same file: 0 mismatched logits at 1e-2 (maxAbs ~2e-5), deterministic, on synthetic input and on a real 10 s log-mel; faithful to the original graph within 4.8e-6 (ORT CPU). Rebuild: capture_shapes.py --frames 513 + rewrite_static_adreno.py in musetric-toolkit scripts/onnx/beat_this/. The host must always feed [1, 513, 128]. Full evidence: the musetric plan gpu/adreno660-strict-2026-09-11.md.

How to use

import * as ort from 'onnxruntime-web/webgpu';

const session = await ort.InferenceSession.create('beat_this.onnx', {
  executionProviders: ['webgpu'],
});

// window: Float32Array(513 * 128), one log-mel chunk (see config.json).
const { beat, downbeat } = await session.run({
  spect: new ort.Tensor('float32', window, [1, 513, 128]),
});

mel-filterbank.bin is a raw row-major float32 matrix, [513, 128] — load it with new Float32Array(await (await fetch(url)).arrayBuffer()) and project magnitudes onto it.

Logits are peak-picked host-side: keep maxima within ±3 frames (peakKernel 7) whose logit exceeds peakThreshold 0, merge peaks ≤ deduplicateWidth frames apart, convert frames to seconds at fps 50, then snap each downbeat to its nearest beat. See the musetric packages/ai host code (runtime/rhythm/, rhythm/beatPeaks.ts) for the full pipeline.

Files

File	Size	SHA256
`beat_this.onnx`	120,259,561 B	`d6b41a44dbf555e90593f60dc86aea3689e1f5db427956e4c9036c8dfde970e8`
`mel-filterbank.bin`	262,656 B	`1ee975d96f44ccf2c3bfe37825c1c1f0b089f5703c7a12a84b1f0a3bce004533`
`config.json`	1,008 B	`46e93c11d7afb78e3eba72cac26e1aced47b9b6558379f928c19b0cf95c9af1d`

mel-filterbank.bin is torchaudio's own MelScale.fb written verbatim, so a host reuses the reference filterbank instead of reimplementing the slaney mel scale. The analysis window is deliberately not shipped: it is a plain periodic Hann, 0.5 - 0.5·cos(2πn/N), and the export refuses to run if the reference window ever stops matching that.

The graph ships Slice where the export emits Split. onnxruntime's WebGPU Split kernel fails to build a compute pipeline on Adreno: the graph loses its very first Split with CreateComputePipelines failed with VK_ERROR_UNKNOWN, the session falls back to the CPU provider, and the tracker still returns beats with nothing in the log naming the cause. A shader that will not compile, not a binding too large — tensor sizes have nothing to say about it. So all 37 fixed-size Split nodes are lowered to one static Slice per output, reading the split sizes off the Constant that fed the node and writing them as starts/ends/axes initializers. Same partition of the same axis, no weight read, 34,463 bytes larger.

Signature — float32 weights, opset ai.onnx 17:

Tensor	Type	Shape	Meaning
`spect` (in)	float32	`[windows, 513, 128]`	one 513-frame log-mel window per call
`beat` (out)	float32	`[windows, 513]`	beat logits; `> 0` is a candidate
`downbeat` (out)	float32	`[windows, 513]`	downbeat logits

Validation

The graph, against the PyTorch File2Beats reference sharing its log-mel and minimal postprocessing, over 20 instrumental stems (the production input, 10 s–285 s):

Metric	Value
tracks with identical beat and downbeat counts	20/20
max beat / downbeat time difference	0.0 s

Beat times are bit-identical: logits agree to ~1e-5 and peak picking quantizes to 20 ms frames, so the residual cannot move a beat.

The WebGPU front end, against torchaudio's LogMelSpect on the same waveforms:

Metric	Value
frame counts	20/20 exact
max absolute difference	3.2e-04 (relative to peak: 3.8e-05)
mean absolute difference	~1e-06

That residual is float32 GPU arithmetic, not a different transform, and it changes nothing downstream: the WebGPU path reproduces the wasm path's results byte for byte on 20/20 tracks.

End to end, the shipped host (ffmpeg mean-downmix decode + WebGPU) against the Python CLI it replaces (torchaudio + soxr), over the same 20 stems:

Metric	Value
`bpm` — the user-visible tempo	20/20 identical
`meter`	20/20 identical
beat F-measure (±70 ms)	mean 0.9982, median 1.0000, min 0.9898
downbeat F-measure (±70 ms)	mean 0.9962, median 1.0000, min 0.9722
beats differing over the whole set	20 missed + 9 extra of 7,853 (0.37 %)

Neither the graph nor the front end causes that, and neither does gain: the mean downmix matches the reference to a ratio of 1.0000 and the two decodes correlate at 0.99998 with zero sample shift. What is left is the resampler — ffmpeg's swr versus soxr — and peak picking turns that into a hard decision, so a handful of marginal beats near the logit > 0 threshold flip. Tempo and meter are medians over hundreds of beats and absorb it. Feeding the reference's own decode through this stack reproduces the CLI exactly, which is what pins the residual on the decoder.

The tables above were measured before the Split → Slice lowering. The two graphs return bit-identical beat and downbeat logits on the CPU provider on the same input, at one window and at three, so the numbers carry over.

Re-run the parity gate on the exact published bytes before relying on it.

Source & lineage

Code license and weight license are separate; ONNX conversion does not change the weight license. Documented only as far as it is verifiable.

Architecture: log-mel front end (n_fft 1024, hop 441, 128 mels, log1p(1000 · x)) → a conv stem and three frontend blocks, each a partial transformer attending across frequency then time → six rotary transformer layers (dim 512, 16 heads) → a sum head emitting beat and downbeat logits. 20.25 M parameters.
Reference implementation and weights: CPJKU/beat_this, MIT (per its LICENSE, Copyright (c) 2024 Institute of Computational Perception, JKU Linz, Austria), consumed as the beat-this package (1.1.0).
Paper: Foscarin, Schlüter, Widmer — Beat this! Accurate beat tracking without DBN postprocessing, ISMIR 2024.
Checkpoint: final0 — beat_this-final0.ckpt, 81,058,141 B, sha256 8c328b45f59d8dd3dff219253ff6a8d6482be57d0133a29140e2febbf8eb8331 — fetched at export time from the upstream checkpoint host via torch.hub, as the package's load_checkpoint does. Upstream serves it from a cloud share rather than a content-addressed URL, so the sha256 above is the pin.
Export tooling: scripts/onnx/beat_this in musetric-toolkit; see its thirdPartyNotices.md.
Host runtime: packages/ai in musetric.

This export preserves the upstream MIT license; we do not claim authorship of the original weights.

License

MIT, inherited from the upstream weights.

Downloads last month: 4,553