Hugging Face's logo

Mothersuperior
/
yue2-mothersuperior-realaudio-tokenizer-v4

audio
music
yue2
tokenizer
lora
Model card Files
xet
 Community
2
yue2-mothersuperior-realaudio-tokenizer-v4 / README.md
Mothersuperior's picture
Mothersuperior
README wording
f2278a2 verified
preview code
|
Raw
History Blame
5.49 kB
metadata 
license: cc-by-nc-4.0
base_model:
  - m-a-p/YuE2-3B
  - m-a-p/MERT-v2-FullSong
tags:
  - audio
  - music
  - yue2
  - tokenizer
  - lora

yue2-mothersuperior-realaudio-tokenizer-v4

Real-audio tooling for YuE2-3B: the audio → semantic-token encoder YuE2 doesn't ship, plus a NAR-branch LoRA so the decoder renders real-production latents. Together they let you tokenize your own recordings, LoRA-tune YuE2's AR on an artist, and generate new songs or covers.

Files

file what
tokenizer_head_joint_v4.pt MERT-v2-FullSong layer-20 features (per-track instance-normalised, 25 Hz) → 32,768 YuE2 semantic codes. 8-layer transformer, d=512, 512-frame windows. Held-out exact match on YuE2's own songs: 16.1% top-1 (near-miss codes render almost identically; ear tests of NAR round-trips sit around 95%).
nar_lora_joint_v4.pt rank-32 LoRA on nar_self_attn.{q,k,v,o}_proj + nar_mlp.{gate,up,down}_proj (28 layers) + full vae2llm/llm2vae. Trained jointly with the head on real audio.
scripts/ the training loop and inference scripts (below).

Trained on 4,765 YuE2 self-generated songs, then adapted to real audio. If your material sounds off, rerun joint.py on your own audio (step 3 below).

Requirements

Python 3.12 venv with yue2-infer (commit 92a73cc7), torch 2.10 + cu128, torchaudio 2.10, transformers, soundfile, scipy, safetensors, demucs; HF_HOME with m-a-p/YuE2-3B, m-a-p/YuE2-Vae, m-a-p/MERT-v2-FullSong. A 24 GB GPU is enough for every stage (14–18 GB measured with gradient checkpointing).

You also need the minted regularizer pack: regularizer/minted_regularizer_pack.pt from Mothersuperior/yue2-minted-corpus (~100 MB, 4,732 YuE2-generated songs as {name, src, style, lyrics, codec}). The AR trainer draws 50% of its songs from it so a small artist set cannot collapse YuE2's token grammar; the minted_val items are the held-out check whose loss should stay flat. The full corpus (audio + tokens + latents) is in the same dataset if you want to retrain the head.

Paths are hard-coded to our pod layout (/workspace/tok/full, /workspace/real/..., /workspace/yue2-corpus/tracks, /workspace/real/ar/dataset.pt). Recreate that layout or edit the constants at the top of each script.

Train an artist LoRA (folder of songs → LoRA)

Per song you need <name>.flac, <name>.lyrics.txt (full lyrics with [verse]/[chorus]/[bridge]/... tags — truncated lyrics ruin structure), and <name>.txt = a style caption starting with your trigger phrase (e.g. xyzq, in the style of xyzq. <description of the sound>).

  1. python prep_real.py — MERT features, VAE latents and the prompt prefix per song.
  2. python cursor_prep.py — Demucs vocal stem → MMS forced alignment of the lyrics → lyric-cursor targets (automatic).
  3. (optional, recommended for a new artist/era) HOLD_TRACK=<one song name> python joint.py joint_mine 3000 1 1 tokenizer_head_joint_v4.pt nar_lora_joint_v4.pt — adapts head + NAR to your audio. Otherwise use the v3 files as-is.
  4. python ar_prep.py <head.pt> — tokenizes your songs and merges the regularizer pack into dataset.pt (ar_prep.py expects the pack's records; point it at the downloaded file).
  5. SCHED_STEPS=3000 CK_FROM=600 CK_EVERY=200 python ar_lora_cursor.py my_lora 1600 64 0.5 none 1e-4 0.08 — rank-64 AR LoRA, 50/50 artist vs minted, lyric-cursor weight 0.08, checkpoints at 600/800/1000/1200/1400/1600. Do not train longer: past ~1,500 steps the model memorises the songs.
  6. LADDER_STYLE_TRACK=<song> LADDER_LYRICS=<lyrics.txt> bash ladder.sh my_lora nar_lora_joint_v4.pt — renders one fixed prompt from every checkpoint (optional FINALS=<file> with lines tag style_track lyrics seed). Pick by ear (ours: step 800).

Decoder

Consider decoding with Mothersuperior/YuE2-Vae-merge-0.666: a weight merge of YuE2-Vae (0.666) and YuE2-Vae-legacy (0.334). The two releases share one encoder and only differ in the decoder; the merge sits between the clean default decoder and the more musical legacy decoder. Drop-in: YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", vae="Mothersuperior/YuE2-Vae-merge-0.666"), or point ar_generate.py at it.

Inference 

# new song: style caption with your trigger + lyrics, score-free
python ar_generate.py my_lora/step-800.pt nar_lora_joint_v4.pt out_tag <style_track> lyrics.txt 12
# cover: transcribe any recording with SheetSage2 (--melody-only), then
ABC_FILE=score.abc COT=melody STRIP_TEMPO_KEY=1 python ar_generate.py my_lora/step-800.pt nar_lora_joint_v4.pt cover_tag <style_track> lyrics.txt 21
# LoRA strength: AR_SCALE=0.77 ...   stock model control: pass `none` for either LoRA

ar_generate.py folds both LoRAs into the base weights and runs YuE2's own pipeline, so the stock sampler, CFG and VAE apply unchanged.

Other scripts

extract_full.py + train_v2.py (retrain the head on the minted corpus), nar_lora.py (NAR LoRA alone), teacher_train.py (head fine-tune with the NAR as teacher), ar_lora.py (AR LoRA without the cursor), build_reg_pack.py (rebuild the regularizer pack from a corpus).

Weights derive from YuE2-3B (CC BY-NC 4.0): non-commercial use only. Scripts are provided as-is.