Mothersuperior commited on
Commit
c2db50d
Β·
verified Β·
1 Parent(s): 18a83ea

rename weights to v4

Browse files
README.md CHANGED
@@ -19,8 +19,8 @@ an artist, and generate new songs or covers.
19
  ## Files
20
  | file | what |
21
  |---|---|
22
- | `tokenizer_head_joint_v3.pt` | MERT-v2-FullSong layer-20 features (per-track instance-normalised, 25 Hz) β†’ 32,768 YuE2 semantic codes. 8-layer transformer, d=512, 512-frame windows. Held-out exact match on YuE2's own songs: 16.1% top-1 (near-miss codes render almost identically; ear tests of NAR round-trips sit around 95%). |
23
- | `nar_lora_joint_v3.pt` | rank-32 LoRA on `nar_self_attn.{q,k,v,o}_proj` + `nar_mlp.{gate,up,down}_proj` (28 layers) + full `vae2llm`/`llm2vae`. Trained jointly with the head on real audio. |
24
  | `scripts/` | the training loop and inference scripts (below). |
25
 
26
  Trained on 4,765 YuE2 self-generated songs, then adapted to real audio. If your material sounds off, rerun `joint.py` on your own audio (step 3 below).
@@ -45,21 +45,21 @@ and `<name>.txt` = a style caption starting with your trigger phrase (e.g. `xyzq
45
 
46
  1. `python prep_real.py` β€” MERT features, VAE latents and the prompt prefix per song.
47
  2. `python cursor_prep.py` β€” Demucs vocal stem β†’ MMS forced alignment of the lyrics β†’ lyric-cursor targets (automatic).
48
- 3. *(optional, recommended for a new artist/era)* `HOLD_TRACK=<one song name> python joint.py joint_mine 3000 1 1 tokenizer_head_joint_v3.pt nar_lora_joint_v3.pt`
49
  β€” adapts head + NAR to your audio. Otherwise use the v3 files as-is.
50
  4. `python ar_prep.py <head.pt>` β€” tokenizes your songs and merges the regularizer pack into `dataset.pt` (`ar_prep.py` expects the pack's records;
51
  point it at the downloaded file).
52
  5. `SCHED_STEPS=3000 CK_FROM=600 CK_EVERY=200 python ar_lora_cursor.py my_lora 1600 64 0.5 none 1e-4 0.08`
53
  β€” rank-64 AR LoRA, 50/50 artist vs minted, lyric-cursor weight 0.08, checkpoints at 600/800/1000/1200/1400/1600. **Do not train longer**: past
54
  ~1,500 steps the model memorises the songs.
55
- 6. `LADDER_STYLE_TRACK=<song> LADDER_LYRICS=<lyrics.txt> bash ladder.sh my_lora nar_lora_joint_v3.pt` β€” renders one fixed prompt from every checkpoint (optional `FINALS=<file>` with lines `tag style_track lyrics seed`). Pick by ear (ours: step 800).
56
 
57
  ## Inference
58
  ```bash
59
  # new song: style caption with your trigger + lyrics, score-free
60
- python ar_generate.py my_lora/step-800.pt nar_lora_joint_v3.pt out_tag <style_track> lyrics.txt 12
61
  # cover: transcribe any recording with SheetSage2 (--melody-only), then
62
- ABC_FILE=score.abc COT=melody STRIP_TEMPO_KEY=1 python ar_generate.py my_lora/step-800.pt nar_lora_joint_v3.pt cover_tag <style_track> lyrics.txt 21
63
  # LoRA strength: AR_SCALE=0.77 ... stock model control: pass `none` for either LoRA
64
  ```
65
  `ar_generate.py` folds both LoRAs into the base weights and runs YuE2's own pipeline, so the stock sampler, CFG and VAE apply unchanged.
 
19
  ## Files
20
  | file | what |
21
  |---|---|
22
+ | `tokenizer_head_joint_v4.pt` | MERT-v2-FullSong layer-20 features (per-track instance-normalised, 25 Hz) β†’ 32,768 YuE2 semantic codes. 8-layer transformer, d=512, 512-frame windows. Held-out exact match on YuE2's own songs: 16.1% top-1 (near-miss codes render almost identically; ear tests of NAR round-trips sit around 95%). |
23
+ | `nar_lora_joint_v4.pt` | rank-32 LoRA on `nar_self_attn.{q,k,v,o}_proj` + `nar_mlp.{gate,up,down}_proj` (28 layers) + full `vae2llm`/`llm2vae`. Trained jointly with the head on real audio. |
24
  | `scripts/` | the training loop and inference scripts (below). |
25
 
26
  Trained on 4,765 YuE2 self-generated songs, then adapted to real audio. If your material sounds off, rerun `joint.py` on your own audio (step 3 below).
 
45
 
46
  1. `python prep_real.py` β€” MERT features, VAE latents and the prompt prefix per song.
47
  2. `python cursor_prep.py` β€” Demucs vocal stem β†’ MMS forced alignment of the lyrics β†’ lyric-cursor targets (automatic).
48
+ 3. *(optional, recommended for a new artist/era)* `HOLD_TRACK=<one song name> python joint.py joint_mine 3000 1 1 tokenizer_head_joint_v4.pt nar_lora_joint_v4.pt`
49
  β€” adapts head + NAR to your audio. Otherwise use the v3 files as-is.
50
  4. `python ar_prep.py <head.pt>` β€” tokenizes your songs and merges the regularizer pack into `dataset.pt` (`ar_prep.py` expects the pack's records;
51
  point it at the downloaded file).
52
  5. `SCHED_STEPS=3000 CK_FROM=600 CK_EVERY=200 python ar_lora_cursor.py my_lora 1600 64 0.5 none 1e-4 0.08`
53
  β€” rank-64 AR LoRA, 50/50 artist vs minted, lyric-cursor weight 0.08, checkpoints at 600/800/1000/1200/1400/1600. **Do not train longer**: past
54
  ~1,500 steps the model memorises the songs.
55
+ 6. `LADDER_STYLE_TRACK=<song> LADDER_LYRICS=<lyrics.txt> bash ladder.sh my_lora nar_lora_joint_v4.pt` β€” renders one fixed prompt from every checkpoint (optional `FINALS=<file>` with lines `tag style_track lyrics seed`). Pick by ear (ours: step 800).
56
 
57
  ## Inference
58
  ```bash
59
  # new song: style caption with your trigger + lyrics, score-free
60
+ python ar_generate.py my_lora/step-800.pt nar_lora_joint_v4.pt out_tag <style_track> lyrics.txt 12
61
  # cover: transcribe any recording with SheetSage2 (--melody-only), then
62
+ ABC_FILE=score.abc COT=melody STRIP_TEMPO_KEY=1 python ar_generate.py my_lora/step-800.pt nar_lora_joint_v4.pt cover_tag <style_track> lyrics.txt 21
63
  # LoRA strength: AR_SCALE=0.77 ... stock model control: pass `none` for either LoRA
64
  ```
65
  `ar_generate.py` folds both LoRAs into the base weights and runs YuE2's own pipeline, so the stock sampler, CFG and VAE apply unchanged.
nar_lora_joint_v3.pt β†’ nar_lora_joint_v4.pt RENAMED
File without changes
tokenizer_head_joint_v4.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d23c4f757a05f031134b8471ec84245ec2338966516e1a9e26a17ff300a5f87e
3
+ size 171305291