README.md

---
license: cc-by-nc-4.0
base_model: m-a-p/YuE2-3B
base_model_relation: quantized
pipeline_tag: text-to-audio
tags:
  - yue2
  - music-generation
  - fp8
  - compressed-tensors
  - vllm
  - dgx-spark
language:
  - en
  - zh
---

# YuE2-3B-FP8 (autoregressive checkpoint, W8A8)

This is an FP8 (W8A8, dynamic per-token activations) requantisation of the autoregressive (AR) half of
[m-a-p/YuE2-3B](https://huggingface.co/m-a-p/YuE2-3B). It is packaged as the
Qwen3-shaped checkpoint that the official `yue2_infer` pipeline derives
internally for its vLLM backend, and it is a drop-in replacement for that
derived checkpoint. The planning (abc) and semantic-token stages run on these
weights. The NAR flow-matching stage and the VAE still run on the original
bf16 and fp32 weights, which you need alongside this repo.

The point of it is decode speed on bandwidth-limited hardware. On a DGX Spark
(GB10, 273 GB/s unified memory) the AR stages are limited by weight bandwidth,
and the bf16 AR checkpoint is 4.33 GB.

| | bf16 (derived by yue2_infer) | this checkpoint |
|---|---|---|
| AR weights on disk | 4.33 GB | 2.92 GB |
| Linear layers | bf16 | FP8 e4m3 weights with a scale per output channel, FP8 e4m3 activations scaled per token at run time |
| embed_tokens and lm_head (vocab 184,704) | bf16 | bf16, not quantised |
| AR decode on DGX Spark, vLLM 0.27.1, batch 1 | 44.8 tok/s abc, 43.0 tok/s semantic | 79 to 81 tok/s abc, 67 to 76 tok/s semantic (1.8x) |

This build is the fidelity option. The NVFP4 sibling
(DKmode22/YuE2-3B-NVFP4) is smaller and faster on unified-memory hardware
at a somewhat larger deviation from bf16; the numbers for both are below.

## What was quantised and how

Source: `model.safetensors` from m-a-p/YuE2-3B (sha256 starting `1d55c42c`).
The AR tensors were extracted exactly as `yue2.fast.derive_ar_checkpoint` does:
311 tensors covering attention q/k/v/o and their q/k norms, the AR MLPs, layer
norms, embed_tokens and lm_head. The NAR MLPs, the llm2vae and vae2llm
projections, the timestep embedder and the VAE are not in this checkpoint and
are not quantised.

Tool: [llm-compressor](https://github.com/vllm-project/llm-compressor) 0.13.0
with compressed-tensors 0.18.0. Recipe:

```
QuantizationModifier(targets=["Linear"], scheme="FP8_DYNAMIC", ignore=["lm_head"])
```

Calibration used sequences the bf16 model generated itself, not a text corpus,
because text never exercises the codec-token activations. The set was 32
requests (16 lyric sets, each rendered under two contrasting style prompts,
English and Mandarin, cot=full and cot=melody), both the abc and semantic
phases, cut into 58 windows of up to 8,192 tokens: 200K tokens in total, 51% of
them codec tokens. Four requests were held out for evaluation.

Format: compressed-tensors `float-quantized`. Loads in vLLM on any GPU with
FP8 support (Ada, Hopper, Blackwell including DGX Spark), and in transformers
with the weights dequantised to bf16.

## Quality against bf16

Measured through vLLM 0.27.1 on the four held-out requests, with every output
position teacher-forced on the bf16 model's own sampled sequences: 6,286 abc
positions and 17,931 semantic positions. The first row is two bf16 runs on
different engine builds, which gives the numerical floor.

Top-1 agreement is a poor guide for the semantic phase. The codec distribution
is nearly flat at most positions (bf16 itself puts its argmax on the token it
sampled only 22% of the time), so a small shift flips the argmax without
moving much probability. The last column, total-variation distance, is the
share of sampling probability that actually moved, and is the number to read.

| Checkpoint | abc top-1 agree | abc NLL | semantic top-1 agree | semantic NLL | semantic TV |
|---|---|---|---|---|---|
| bf16 vs bf16, two engine builds | 0.999 | 0.141 / 0.141 | 0.982 | 3.414 / 3.414 | 1.2% |
| this checkpoint, FP8 W8A8 | 0.997 | 0.141 / 0.141 | 0.942 | 3.414 / 3.418 (+0.1%) | 3.8% |
| NVFP4 W4A16 GPTQ, sibling repo DKmode22/YuE2-3B-NVFP4 | 0.996 | 0.141 / 0.142 | 0.905 | 3.414 / 3.424 (+0.3%) | 6.3% |
| NVFP4 W4A16, AWQ | 0.993 | 0.141 / 0.143 | 0.896 | 3.414 / 3.433 (+0.6%) | 7.4% |
| NVFP4 W4A4, GPTQ (not published) | 0.992 | 0.141 / 0.144 | 0.854 | 3.414 / 3.446 (+0.9%) | 10.1% |

Split by how confident bf16 was at each position, both published checkpoints
agree with bf16 at more than 99.8% of positions where the bf16 top-1
probability is above 0.5; the disagreements sit in near-tie positions. Streams
were checked for repetition: no repeated 24-gram on any held-out output.

## How to use

The official pipeline derives its own bf16 AR checkpoint and does not yet accept
an alternative. `patch_fast.py` in this repo adds a `YUE2_AR_CHECKPOINT`
environment override to `yue2/fast.py`. It is idempotent and changes nothing
else.

```bash
pip install "yue2_infer[fast]"        # the wheel from m-a-p/YuE2-3B
python patch_fast.py "$(python -c 'import yue2.fast,os;print(os.path.abspath(yue2.fast.__file__))')"
hf download DKmode22/YuE2-3B-FP8 --local-dir ./yue2-3b-fp8
export YUE2_AR_CHECKPOINT=$PWD/yue2-3b-fp8
python - <<'EOF'
from yue2 import YuE2Pipeline
pipe = YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", backend="vllm", memory_budget_gib=16)
song = pipe(style="Indie folk, warm acoustic guitar, male vocal", lyrics="[Verse]\n...")
song.save("song.flac")
EOF
```

Notes:

- The pipeline uses vLLM only when cfg_scale is 1 (the default for cot=full and
  cot=melody) and quantization is "none". With cot=off the guidance is 1.01 and
  the pipeline silently falls back to its torch backend, which does not use
  this checkpoint.
- Measured on DGX Spark (sm_121) with vLLM 0.27.1. The package pins
  `vllm==0.19.0`; to use a newer build, widen the version check in
  `yue2/fast.py`. The 0.19 wheel's FP8 path was not tested on sm_121.
- Everything else in the pipeline is unchanged: one sequence at a time, the
  windowed repetition penalty, 24,576 context.

## Licence and attribution

The base model is released by the M-A-P team under CC-BY-NC-4.0. This
derivative carries the same licence and is for non-commercial use. Credit for
the model belongs to the YuE2 authors; please cite their paper
(arXiv 2503.08638). Quantisation, calibration corpus and evaluation by
DKmode22 on DGX Spark hardware, September 2026.