How these files were checked
What these files are
ACE-Step Captioner is the ACE-Step team's fully fine-tuned Qwen2.5-Omni-7B thinker for music description, released under MIT. Side-Step converted it to GGUF for llama.cpp.
The download has a thinker file and an audio projector. The projector contains the audio encoder. The published pair is acestep-captioner-Q4_K_M.gguf at 4.7 GB and acestep-captioner-mmproj-Q8_0.gguf at 1.5 GB.
All checks used the prompt *Task* Describe this audio in detail and greedy decoding. The evaluation used 23 one-minute instrumental pop excerpts from the Song Describer Dataset, which is not redistributed.
Why the engine matters
The GGUF files need a particular llama.cpp engine overlay. Stock llama.cpp b10796 loaded them, but two faults made its captions unacceptable for training.
First, stock llama.cpp added 30 seconds of silence to every input. At exact 30-second multiples, that created a complete extra piece. A 60-second clip became three pieces and 2,264 prompt tokens instead of two pieces and 1,514. Across four GGUF builds, 92 of 92 captions mentioned a cut to silence. The original did so in 3 of 23. The overlay drops the extra piece. Silence mentions then fell to 0 to 3 of 23. Other input lengths did not trigger this fault.
Second, the stock audio encoder used one long attention range, one continuous convolution, and one position count. The original uses 2-second blocks, separate 200-frame chunks, and restarts position counting. Agreement was 0.32 on stock and 0.997 with all corrections. The early GGUF variants failed alike, while NF4 in transformers passed.
The Side-Step engine overlay contains these corrections. They are recorded as two MIT-licensed patches in the engine repository. On Windows, it needs the Microsoft Visual C++ 2015-2022 Redistributable.
How the files were checked
The training check asks whether a caption can still guide ACE-Step training. The frozen ACE-Step decoder ran in bf16 on CUDA, with its final comparison in fp32. For each of 23 clips and five diffusion timesteps, own-caption loss was compared with eight captions from other clips. Those wrong-caption losses show how much captions can matter. The int8 original was the reference, and it matched the bf16 original. The protocol was frozen and hashed before results were calculated.
There were two gates. Fidelity checks right-caption loss. Discrimination checks right and wrong caption separation. Each allows a quarter of the wrong-caption spread plus two standard errors. Both must pass at 0.7, 0.9, and 0.96. At 0.3 and 0.5, losses were too close to distinguish captions, so they were reported but not gated.
For six clips, captions were shown in random order without engine names. Four pairs were judged the same. GGUF was preferred twice and the original never. One original caption contradicted the audio. This shows the port was not worse than the original, not absolute accuracy.
What passed
Against the int8 original, the bf16 original, NF4 in transformers, and the published Q4_K_M + Q8_0 pair passed every gated check. Context 2048 and flash attention produced byte-identical captions, so they inherit the matching result.
The default is Q4_K_M with the Q8_0 projector. It is the smallest passing llama.cpp pair.
The same Q4_K_M file with Patch 1's earlier encoder failed the training check at timestep 0.7, at 1.10 times the tolerance. The earlier engine could load these files, but its captions failed the training check and described silence for inputs at exact 30-second multiples.
What it costs to run
These measurements used CUDA on Windows 11 with an RTX 4070 Ti SUPER 16 GB, 23 clips of 60 seconds, greedy decoding, and one request at a time. The desktop adds its own use to the whole-card figure.
| build | engine | captioner's own VRAM peak | whole card, desktop included | seconds per clip | load |
|---|---|---|---|---|---|
| Q4_K_M + Q8_0 projector, ctx 4096 | llama.cpp | 6.9 GiB | 7.8 GiB with a 0.9 GiB desktop; 8.2 GiB with a 1.3 GiB desktop | 1.4 to 1.5 | 4 s |
| Q4_K_M + Q8_0 projector, ctx 2048 | llama.cpp | 6.8 GiB | 8.1 GiB with a 1.3 GiB desktop | 1.5 | 4 s |
| original, NF4 | transformers | 8.4 GiB allocated | not measured | 6.3 | 14 s |
| original, int8 | transformers | 11.6 GiB allocated | not measured | 16.7 | 13 s |
| original, bf16, partly on CPU | transformers | 13.4 GiB allocated | not measured | 51.2 | 18 s |
| tier | configuration | captioner's own VRAM peak | whole card with a 1.3 GiB desktop | host RAM peak | seconds per clip | 300 clips | passed the training check |
|---|---|---|---|---|---|---|---|
| 12 GB and up | all 29 layers on GPU | 6.8 GiB | 8.1 GiB | 6.6 GiB | 1.5 | 7 min | yes, byte-identical to the default |
| 8 GB | 20 of 29 layers on GPU | 5.6 GiB | 6.9 GiB | 6.6 GiB | 5.4 | 27 min | yes |
| 6 GB | 4 of 29 layers on GPU | 3.5 GiB | 5.0 GiB | 6.7 GiB | 12.0 | 60 min | yes |
The audio encoder stays on the GPU in every tier. Moving it to the CPU saves about 2 GiB more but makes the audio step roughly fifty times slower, so the tiers move thinker layers instead. At four layers, more than half of the GPU memory is the audio encoder and its working buffers, which is the practical floor for this captioner. Running four clips at once on the 6 GB tier was 24.7 % faster and missed the 25 % bar set in advance, so it is not offered.
What was not checked
The work did not measure absolute caption accuracy against human ground truth, audio beyond the 23 excerpts, another GPU or operating system, or a full 300-clip corpus run. The 300-clip figures are 23-clip averages multiplied out.
For the detailed audit trail, see the admission report that ships with the Side-Step source repository. Its protocol appears first and is protected by a hash.
Glossary for the raw report
reference / int8-fresh: The original captioner's newly generated int8 captions, used as the comparison baseline.
donor: A caption borrowed from a different clip to make a deliberately wrong-caption comparison.
spread: The average variation among the eight wrong-caption losses for a timestep.
d_fit and d_disc: The paired difference from the reference for right-caption loss and right-versus-wrong separation.
G1 and G2: The fidelity and discrimination gates used to decide whether a candidate is admitted.
SE: Standard error, an estimate of uncertainty from the 23 paired clip results.
resolution: The smallest average loss difference that the bf16 measurement can reliably show.
t: A diffusion timestep at which the training check is measured.
admitted: Passed every required gate under the frozen protocol.
inherited / byte-identical: Kept a result because every generated caption was identical to a configuration already admitted.
patched, patched2, patched3: Stock plus the silence fix; then plus the attention mask; then plus chunked convolution and position restart.
S14: The stage that made the fresh original-captioner reference and initial transformer results.
S16: The stage that found and fixed the extra silent chunk.
S18: The stage that confirmed the missing attention mask and added it.
S19: The stage that matched chunked convolution and position restart, then checked the bf16 reference chain.
S20: The stage that measured each memory-saving setting on its own.
S21: The stage that chose the 8 GB and 6 GB tiers by moving thinker layers to the CPU instead of the audio encoder.
A-rows, B-rows and T-tiers in S20 and S21: Single memory-saving changes, then the combined configurations selected from them.
thinker: The main language model file that writes the description.
projector / mmproj: The file that holds the audio encoder and passes its output to the thinker.
ngl: The number of thinker layers placed on the GPU.
ctx: The context size, or the amount of prompt and audio-token space available to the engine.