livechord-beat-refiner

Bidirectional Transformer that refines a prior beat tracker's output (beat_this, madmom, or librosa) using full-context audio features. Drops in as a post-processing pass to fix phase drift and bar misalignment without re-running the expensive front-end model.

🎹 Try it live

This model powers the beat / bar grid behind the chord ribbon, 88-key waterfall, and AI accompaniment at livechord.org — upload an MP3 and play along on a virtual piano with chord cards that follow the music.

Quickstart

pip install livechord-beat-refiner

from livechord_beat_refiner import refine

# 1. Run any beat tracker (beat_this, madmom, librosa, ...) → beats / downbeats lists
# 2. Pass them to refine() together with the audio file:
out = refine(
    audio_path="song.flac",
    beats=[0.0, 0.52, 1.05, 1.57, ...],        # seconds
    downbeats=[0.0, 2.10, 4.21, ...],          # seconds
)

print(out["refined_beats"])       # list[float]  — refined beat times
print(out["refined_downbeats"])   # list[float]
print(out["applied"])             # bool         — False if model couldn't run; input echoed back

The checkpoint is downloaded from this Hub repo on first call (cached under ~/.cache/huggingface/hub). To use a local file, pass checkpoint_path="path/to/model.safetensors".

What the model does

audio (22050 Hz, mono)
   │
   ▼
[ CQT 84 + chroma 12 + onset + RMS ]                  ← 98 audio channels
[ initial beat grid ]                                  ← 1 channel  (from prior tracker)
[ initial downbeat grid ]                              ← 1 channel
   │ concat → (T, 100)
   ▼
[ Linear(100→256) + sinusoidal pos enc ]
   │
   ▼
[ TransformerEncoder × 6 (d=256, h=4, ffn=512, GELU, pre-LN) ]
   │
   ▼              ┌────────────┐
   ├──────────────┤ beat head  ├──→ sigmoid → peak-pick → beat times (s)
   │              └────────────┘
   │              ┌────────────┐
   ├──────────────┤ db head    ├──→ sigmoid → peak-pick → downbeat times (s)
   │              └────────────┘
   │              ┌────────────┐
   └──────────────┤ cb head    ├──→ aux supervision (not production-ready, see Limitations)
                  └────────────┘

• Parameters: ≈ 3.0 M
• Frame rate: 10.766 fps (sr=22050, hop=2048)
• Max audio length: ≈ 15 min (longer truncated)
• Inference budget: ~1.8 s per minute of audio on CPU; ~0.3 s/min on a recent NVIDIA GPU.

Why a refiner

State-of-the-art beat trackers (beat_this, madmom) are strong on percussive genres but drift in three failure modes that show up over and over in real-world libraries:

1. Phase shift on slow ballads — the tracker locks onto an offbeat or jumps half a beat partway through.
2. Doubletime / halftime confusion — pop ballads tracked at 138 BPM that are actually 69 BPM.
3. Bar misalignment — beats are right, downbeats off by one.

Re-running the front-end tracker rarely helps (same audio, same answer). The refiner sees both the audio AND the prior tracker's grid as input hints, then re-emits a cleaner grid by attending across the full song with chord-boundary cues as auxiliary supervision.

Metrics

Held-out test set: 7,389 songs (15% stratified holdout from a 13,017-song corpus). Tolerance: ±1 frame ≈ ±93 ms (mir_eval-style F-measure; standard is 70 ms).

Headline (gold-quality subset, n=510)

Songs whose prior-tracker output passed coefficient-of-variation < 0.05 AND chord-change alignment ≥ 0.5 filters — i.e. the most reliable references.

Metric	F1	Precision	Recall
Beat	0.920	0.910	0.933
Downbeat	0.936	0.925	0.952

Full holdout (n=7,389)

Metric	F1	Precision	Recall
Beat	0.881	0.859	0.914
Downbeat	0.914	0.906	0.930

The "skip" quality bucket (6,087 / 7,389 songs) consists of tracks whose prior-tracker reference didn't pass the gold/ok filters, so their "labels" are noisy by construction. The lower full-holdout F1 reflects label noise, not model regression — quality is monotonic (gold > ok > skip), which is the structural sanity check.

Per-bucket

beat_quality	n	beat F1	downbeat F1
gold	510	0.920	0.936
ok	792	0.906	0.925
skip	6,087	0.874	0.910

Reproduce with the eval script in the LiveChord repo.

Note on baseline

The training corpus uses each song's own prior-tracker output as the supervision target whenever that output passed the filter. So the F1 reported here measures how well the refiner preserves correct beats while also absorbing chord-boundary auxiliary signal — not "refiner > beat_this" head to head. An independent ground-truth comparison (MIREX Beatles, GTZAN_rhythm) is on the v2 roadmap.

Training data

• Corpus: 13,017 songs from a personal music library, predominantly East Asian pop, Western pop / rock / R&B, jazz standards, and a smaller classical / folk tail.
• Labels: prior-tracker output (beat_this final0 checkpoint) filtered to gold / ok quality bands by coefficient-of-variation and chord-alignment heuristics.
• Split: 70% train / 15% val / 15% test, stratified by (beat_quality × chord_quality).
• Augmentation: random initial-grid corruption during training (jitter / drop / insert / phase-shift) so the model learns to denoise rather than copy the input grid verbatim.

Limitations

• Chord-boundary head is auxiliary supervision, not a production target. cb F1 on gold = 0.243. v1 didn't tune class weights or peak-picking thresholds for this head; treat the cb output as exploratory.
• No genre rebalancing. Folk / classical / EDM are underrepresented vs pop; performance there is more variable.
• Long songs (> 15 min) are truncated. v1 has no chunked-overlap inference. If you need this, please file a GitHub issue.
• Label noise. Because labels are filtered prior-tracker output, the model inherits any systematic biases that survived the filter.

How to use with beat_this

beat_this (CPJKU 2024) is a strong front-end choice, especially for percussive genres. Pipeline:

from beat_this.inference import File2Beats
from livechord_beat_refiner import refine

# Front-end (GPU-friendly, run once per song)
predictor = File2Beats(checkpoint_path="final0", device="cuda", float16=True)
init_beats, init_downbeats = predictor("song.flac")

# Refine
out = refine(
    audio_path="song.flac",
    beats=init_beats,
    downbeats=init_downbeats,
)
final_beats, final_downbeats = out["refined_beats"], out["refined_downbeats"]

Citation

@misc{livechord-beat-refiner,
  title  = {livechord-beat-refiner: a bidirectional Transformer for
            beat / downbeat / chord-boundary refinement},
  author = {LiveChord Project},
  year   = {2026},
  url    = {https://huggingface.co/livechord-music/livechord-beat-refiner},
  note   = {Refines beat\_this / madmom / librosa output using full
            audio context. Trained on 13,017 songs.},
}

License

Apache License 2.0 — code AND weights. The training data is not redistributed; only the trained model artifact is released.

The LiveChord product (full FastAPI server, frontend, and AI pipeline) is released separately under AGPL v3 at github.com/JJ110112/LiveChord. This package is the standalone inference release.

Related

• livechord-bar-arbitrator — companion phase-correction post-processor that runs after this model to fix bar / beats-per-bar / doubletime confusion using chords[] as an additional signal.
• CPJKU/beat_this — recommended upstream beat tracker.
• livechord.org — the live application this model powers.