---
license: mit
tags:
  - audio
  - music
  - minimax-music3
  - encoder
  - covers
---

# rec7 — an audio encoder for MiniMax-Music3

MiniMax-Music3 ships a language model that emits music codes and hidden
states, a depth decoder, a condition encoder and a flow-matching renderer —
but **not** the audio→codes quantizer. Nothing in the release lets you put
a real recording *into* the model.

rec7 is a substitute for that missing piece. It reads a recording and
produces, per 40 ms frame, the 4096-d hidden state the language model
would have produced for that frame. From those states everything else
follows deterministically using the released components: the 8 code
streams, and the full condition the renderer expects.

What that enables, today:

- **Covers.** Real recording in → M3 renders it back. Timbre, guitar
  articulation and performance come through; the language model is
  skipped entirely. On acoustic singer-songwriter material the result is
  close to indistinguishable from the pipeline's own output quality.
- **Supervision signal.** Hidden states read from real audio are the one
  thing the released M3 never had. They are usable as training targets
  for anything that consumes those states.
- **Reading codes off real audio** (with an important caveat below).

## Quick start

```
pip install -r requirements.txt
# weights/trunk.pt and weights/head.pt come with this repo
# you also need the MiniMax-Music3 checkpoint directory (not included)

# encode: states + codes for a recording
python rec7_encode.py song.mp3 --m3 /path/to/minimax_music3

# cover: render the recording back through M3
python rec7_render.py song.mp3 --m3 /path/to/minimax_music3 \
    --lyrics lyrics.txt --seconds 60 --out cover.flac
```

`dav_loader.py` (a thin loader for the Flow-VAE in `dav.pth`) ships
alongside. The 8B language model is never loaded — the two 16k-row tables
it needs are sliced straight out of the safetensors.

Memory: encoding needs ~3 GB. Covers need the full renderer (~10 GB in
bf16).

## Using the outputs

`rec7_encode.py` writes a `.pt` with `h` `[T, 4096]` (float16), `codes`
`[T, 8]` (int16: c0 is 16384-way, c1–c7 are 1024-way), and optionally
`cond` `[T, 32768]` — the renderer condition — and `z`, the Flow-VAE
latents. 25 frames per second.

To drive the renderer yourself, `rec7_model.states_to_streams()` turns
states into the condition, and `rec7_render.render_with_condition()`
injects it into the pipeline (it patches the semantic generation step to
hand over the states instead of running the LM).

If you build on the states, the frame→latent geometry matters:
`rec7_model.frame_latent_starts()` is the exact piecewise mapping the
renderer uses (200-frame windows on a 100-frame hop, stitched at a
345-latent hop). A constant ratio is wrong by up to a frame and it shows.

## How it was made

- Architecture: conv stem + dilated residual blocks on the Flow-VAE
  latents (86 Hz), exact-span mean-pool to 25 Hz, 8-layer transformer
  (d=1088), a 2-layer head to 4096. ~170M parameters.
- Bootstrapped by self-distillation: ~56k tracks generated by M3 itself,
  storing codes, latents and audio, with the LM's true hidden states dumped
  by teacher-forcing.
- Then trained on non-synthetic audio (~22k files) with a reconstruction objective: states →
  released depth chain → renderer denoising step, judged against the
  recording's own latents. Anchored throughout by supervised code
  prediction on the synthetic pool so the code head doesn't drift.

## Measured performance

| | value |
|---|---|
| held-out code accuracy (c0, synthetic) | 55.7% top-1 |
| state agreement vs. true LM states (synthetic) | 0.88 cosine |
| cover of a dense rock master: content tracking | 0.95 |
| cover: spectral clarity vs. original | 90% |
| cover: transient sharpness vs. original | 127% (over-sharp) |
| cover: high band (5–12 kHz) energy vs. original | 81–84% |

The Flow-VAE round-trip itself preserves ~64% of 5–12 kHz energy on the
same material, so the encoder slightly *overshoots* the high band and adds
some grain — renders read a touch bright and hyped rather than dull. On
sparse acoustic recordings this is much less audible than the numbers
suggest.

## Limitations — read these

**Its codes are not plausible language-model sequences.** Teacher-forced
through the base LM, rec7's code sequences score ~6.2 nats per frame,
against ~2.1 for the model's own samples and ~3.2–3.5 for an encoder
trained to regress the LM's states directly. The states render
beautifully — the renderer doesn't care about sequence plausibility — but
**do not use rec7's codes as training targets for the language model.**
Two fine-tunes trained that way collapsed into droning above modest
adapter strength. Use the states; get codes elsewhere.

**Timbre does not transfer through code-level training.** Fine-tuning the
LM (LoRA or full) on codes teaches composition, phrasing and articulation
and leaves the voice generic. The voice lives in the hidden states, which
code-level losses never touch.

**Trained on 44.1 kHz stereo, 40 ms frames.** Mono is duplicated. Very
short inputs (< ~6 s) are rejected.


## Training

See rec7_train.py for training script. 

## What we found the states are good for

The LM's own hidden states and rec7's states for the same song are related
by an almost linear transform — a fitted 4096×4096 map plus offset lifts
held-out agreement from 0.28 to 0.98, and a constant offset alone gets
0.86. Rendering the LM's teacher-forced states through that map produces
the *recording's singer*, where the raw states produce a generic one.
That is a route to voice identity without touching the language model;
details and tooling to follow.

## Files

```
weights/trunk.pt   the encoder (EMA weights)
weights/head.pt    the state head
rec7_model.py      architecture, geometry, loading, inference helpers
rec7_encode.py     audio -> states/codes
rec7_render.py     audio -> M3 cover
dav_loader.py      Flow-VAE loader
```