YuE2: How Symbolic Planning Makes AI Music Generation Controllable and Editable

From Black Box to Sheet Music: A New Approach to AI Music Generation

Most music generation models treat composition as a black-box operation: you input a text prompt, and audio emerges from a dense neural network—nobody, including the model itself, can inspect or edit the process midway. YuE2 takes a completely different path. According to the technical report released by the multimodal art projection team, it first generates a readable musical score, then renders that score into audio.

The core contribution of this design is what the authors call symbolic planning—a staged generation pipeline that mimics how human composers work. Instead of directly mapping text prompts to audio tokens, the model first writes a structured score in ABC notation (a compact text format that can encode melody, chords, key, time signature, tempo, and song structure), and only then moves to the audio synthesis stage.

This ordering brings two key benefits. First, the symbolic plan acts as a commitment mechanism: harmonic and melodic structure are already fixed, so the model cannot contradict itself midway through a song. Second, the plan is human-readable and machine-editable, which opens up a class of interactions that pure audio models cannot support.

ABC notation is a plain-text music notation format that uses letters to represent pitches (e.g., C, D, E), and numbers and symbols to represent duration, key, time signature, and more. It originated in the 1990s in internet folk music communities, and because it is compact, easy to read and write, and simple for programs to parse, it is widely used in folk tune libraries and algorithmic composition. Compared with MIDI, ABC is organized in text lines and can be directly read, written, and edited by language models; compared with MusicXML, it is more concise and better suited to sequence modeling. In YuE2's pipeline, the ABC score plays the role of an "intermediate representation": the symbolic sequence generated by the model can be read directly by humans, and can also be parsed by downstream programs into note events, which are then sent to the acoustic synthesis stage—thereby decomposing otherwise opaque audio generation into reviewable steps.

Architecture Breakdown: The AR–NAR Hybrid Transformer

Under the hood, YuE2 uses a 28-layer Mixture-of-Transformers (MoT) architecture that runs two expert types in parallel—autoregressive (AR) and non-autoregressive (NAR)—which share attention computation but use separate normalization, projection, and feedforward layers.

The generation process is broken down into three representations:

1. Symbolic tokens (s): sparse sequences encoding melody, chords, key, time signature, tempo, and form—i.e., the ABC score.
2. Semantic music tokens (c): dense 25 Hz sequences that bridge the abstract score and the acoustic signal.
3. Continuous acoustic latents (z): 25 Hz sequences decoded by a variational autoencoder into 48 kHz stereo audio.

The AR component processes symbolic and semantic sequences causally (left to right). The NAR component realizes the acoustic sequence through bidirectional flow matching, enabling it to condition on the full symbolic context when generating audio. The joint probability is expressed as:

p(s, c, z | y) = p_AR(s, c | y) × p_NAR(z | y, s, c)

where y is the text prompt (lyrics, style description, or both).

Autoregressive (AR) generation means the model predicts the next token one by one from left to right, with each step depending on everything generated before it—naturally suited to sequences with temporal order. Non-autoregressive (NAR) generation, by contrast, generates the entire sequence in parallel at once, or iteratively refines it based on full context; it is faster but struggles to directly model strong causal dependencies. YuE2's hybrid design divides the work between the two: structural symbols such as melody and harmony, along with semantic tokens, are generated with AR to ensure temporal consistency; continuous acoustic latents are generated with NAR so they can simultaneously reference the complete symbolic context. Flow matching is a popular generative modeling method in recent years that generates samples by learning a continuous mapping from a simple distribution to the target distribution; compared with diffusion models, it offers more stable training and more efficient sampling, and YuE2 uses it to achieve bidirectional generation at the acoustic layer.

A Training Approach Without Aligned Scores: MERT2 and SheetSage2

This approach faces a practical challenge: most audio training data does not come with aligned symbolic annotations. The team solved this by building two auxiliary models.

SheetSage2 performs full-song lead-sheet transcription, converting audio recordings into ABC notation. During training, it provides the symbolic supervision target s, allowing the model to learn from a large amount of existing music without manually annotated scores.

MERT2 is a multi-view music representation encoder that provides semantic tokens c. It achieved state-of-the-art results on 14 of 15 tasks in the MARBLE music understanding benchmark, providing a solid foundation for the semantic layer.

Both models are released alongside YuE2, meaning the full pipeline—transcription, semantic encoding, and generation—is available for researchers to build on.

Lead-sheet transcription refers to extracting the main melody and chord progression from audio to produce a simplified score (typically containing only the melody line, chord symbols, and basic structure), without capturing full orchestration details. It differs from "full-score transcription," which requires identifying all instrumental parts and is far more difficult. SheetSage2 performs only lead-sheet-level transcription, enabling the model to automatically obtain symbolic supervision signals from large numbers of ordinary recordings and bypassing the manual annotation bottleneck. The MARBLE benchmark is a comprehensive test suite for evaluating music representation learning models, covering more than a dozen tasks including chord recognition, beat tracking, key detection, and genre classification, used to measure whether an encoder has learned general-purpose music understanding. MERT2 being best on 14 of 15 tasks indicates that its semantic tokens can carry music information shared across tasks, providing a reliable supervision target for YuE2's semantic layer.

What Symbolic Planning Enables

The ABC score is not just an internal implementation detail—it is an explicit output that users and downstream systems can interact with.

Agentic music editing: Because the score is text, a language model can translate natural-language feedback (such as "make the chorus more upbeat" or "change the key to D minor") into concrete score revisions. YuE2 then re-renders the modified score into audio while preserving the unedited parts. This creates a true editing loop rather than a generate-and-discard workflow.

Zero-shot cover songs: SheetSage2 can transcribe a reference recording into ABC notation. YuE2 can then apply a new style prompt while preserving the melodic structure, generating a style-transferred cover without any fine-tuning.

Targeted local editing: Because the score is structured, users can modify a single measure—changing the chord configuration or a melodic phrase—and re-render only the affected portion. Pure audio models have no equivalent mechanism.

Performance: Competing with Commercial Systems

On the WildSongBench (WSB) benchmark, YuE2 achieved a SongBench global average score of 6.73, the highest among all public baselines at the time of the report's release. With a best-of-8 selection strategy, the score rose to 6.96—the highest mean observed across all evaluated systems, including commercial ones.

In expert listening evaluations, participants preferred symbolic-planning outputs over non-planned generations from the same checkpoint at a rate of 49.3% versus 34.6% (the remainder indicating no preference). When compared directly with Suno v5, the best-of-8 configuration produced nearly balanced preference—a notable result for an open-weight system competing with a closed-source commercial product.

The best-of-8 selection strategy means the model generates 8 candidate outputs for the same prompt, then uses a scoring model or manual rules to pick the best one. This practice can significantly boost evaluation scores because extremely good samples are more likely to appear among multiple draws, but the cost is roughly an 8x increase in inference cost. Therefore, comparisons should be read with care: a system using best-of-8 is not entirely fairly compared against a system using single-sample generation. Preference rate comes from blind listening tests, in which judges choose which audio they prefer without knowing its source; 49.3% versus 34.6% means the remaining roughly 16% expressed no preference. A value close to 50% indicates that the two are hard to distinguish, and it is a commonly used reference metric for open-source systems benchmarking against closed-source commercial products.

Implications for Developers

For developers building music-related applications, YuE2 offers something most generative audio models lack: a structured intermediate representation that can be inspected, validated, and programmatically modified. If you need to enforce harmonic constraints, ensure a specific song structure, or integrate user feedback into the generation loop, the ABC score provides a natural intervention point.

The model weights, code, and two auxiliary models (MERT2 and SheetSage2) have all been released on GitHub and Hugging Face. The project page map-yue2.github.io contains audio examples demonstrating cover and editing workflows.

The Long-Term Value of Structured Transparency

YuE2's contribution is not just a better music generator—it demonstrates that incorporating an explicit, inspectable intermediate representation into the generation pipeline can improve both quality and controllability at the same time. The symbolic planning approach trades some generation speed for a system in which composers, developers, and AI agents can genuinely collaborate, rather than simply prompt and pray. As music generation matures from a novelty into a production tool, this kind of structured transparency may become more important, not less.

One-line Summary

YuE2 enables transparent, controllable music generation by first generating an editable ABC score and then synthesizing audio.

Paragraph Summary

YuE2 proposes a staged music generation approach called "symbolic planning": instead of directly mapping text prompts to audio, the model first writes a structured score in ABC notation, then renders audio from that score. Its architecture uses a 28-layer Mixture-of-Transformers that runs autoregressive and non-autoregressive experts in parallel, decomposing generation into three layers—symbolic tokens, semantic tokens, and continuous acoustic latents—and uses flow matching for acoustic synthesis. To address the lack of training data with aligned scores, the team built two auxiliary models: SheetSage2 (audio-to-score) and MERT2 (semantic encoding). The symbolic plan provides an inspectable, editable intermediate representation supporting agentic editing, zero-shot covers, and local modifications; it achieved a 6.73 global average on WildSongBench, rising to 6.96 under best-of-8, with preference close to Suno v5.