ComfyUI-Olm-YuE2: A Stage-by-Stage Controllable AI Music Generation Workflow

Recently, the music generation model YuE2 was officially released, opening new possibilities for AI music creation. Developer o-l-l-i built a custom node suite called ComfyUI-Olm-YuE2 on top of it, deeply integrating YuE2 into ComfyUI workflows. Unlike most "one-click generation" wrappers on the market, this project deliberately exposes more of YuE2's internal intermediate steps, giving creators a genuine understanding of — and control over — every stage of the music generation process.

From "One-Click Generation" to Stage-by-Stage Controllable Generation

In his Reddit post, the author admitted that he originally just wanted to get YuE2 "running properly" inside ComfyUI — only to find himself spending far more time than expected diving deep into the internals. The result of that over-investment is an integration that goes far beyond a simple wrapper.

That said, basic usage remains straightforward: just provide a music style description and lyrics to generate a 48 kHz stereo track. But what truly sets this project apart is how it splits the generation pipeline into four clearly defined stages:

Plan → Semantic → Synthesize → Decode

The intermediate output of each stage can be inspected, saved, reused, or branched. This means you're no longer tossing lyrics into a black box and waiting for results — instead, you can observe and adjust the music generation process step by step within a standard ComfyUI workflow, much like debugging code.

Why "Exposing Internal Steps" Matters

The author explicitly states that the design intent was "for learning" — he wanted the integration to reveal what makes YuE2 interesting, rather than hiding everything behind a single Generate button. For users who want to deeply understand how AI music generation works, this decomposable, observable architecture is far more valuable than a convenient but opaque end-user tool.

YuE2's four-stage pipeline maps to the hierarchical architecture common in modern music generation models: the Plan stage generates a high-level musical structure blueprint, covering tonality and section arrangement; the Semantic stage converts text and musical planning into discrete semantic token sequences that capture melodic contours and harmonic outlines; the Synthesize stage generates fine-grained acoustic tokens from the semantic tokens, determining timbre and expressive detail; and the Decode stage converts acoustic tokens back into playable audio waveforms. This layered design stems from combining language models with audio codecs (such as EnCodec), following a similar approach to Meta's MusicGen. By exposing intermediate outputs at each layer, users can intervene at different levels of abstraction without rerunning the entire compute-intensive pipeline from scratch.

Editable Scores: ABC Notation and Branching Creation

The project's most distinctive feature is an optional score editing UI (enabled in ComfyUI settings). It includes several notably professional components:

• Rendered staff notation in the ComfyUI sidebar
• Raw ABC notation view
• An ABC editor node
• A workflow where you generate a plan, edit the score, then continue generation from the modified version

The significance of this workflow is that it breaks through the traditional pain point of AI music generation being "uncontrollable." You can let the model generate a musical plan, manually edit the melody or structure using ABC notation, then continue the generation pipeline from your modified version. This "human-in-the-loop" approach transforms AI from a pure generator into a collaborative instrument.

It's worth emphasizing that the score UI is entirely optional. Standard node execution and API workflows don't depend on it, preserving flexibility for automated scenarios.

ABC notation is a plain-text music format originally designed by Chris Walshaw in the 1980s for folk music. It uses letters A–G for pitches, numbers for durations, and symbols for accidentals — for example, A2 B c | d e f g fully describes a melodic phrase. Compared to MIDI or MusicXML, ABC notation is human-readable and easy to edit by hand, and is widely used in folk and traditional music communities. Its advantage in AI music generation pipelines is that model-generated musical plans can be presented directly to users as text, allowing basic melodic edits — such as changing pitches, adjusting rhythm, or swapping out a phrase — without requiring formal music theory knowledge. Edited results can then be fed back into the downstream synthesis and decode stages for fine-grained control over the output.

Engineering Details: Minimal Dependencies and Environment Compatibility

Anyone familiar with the ComfyUI ecosystem knows that dependency conflicts are often the biggest headache when installing third-party nodes. The author put considerable effort into this, adopting a notably conservative strategy:

It reuses ComfyUI's existing Torch/Transformers rather than installing YuE2's officially pinned dependency stack — only tiktoken and accelerate need to be added on top of a clean ComfyUI environment.

This "don't fight with the existing environment" philosophy dramatically lowers the installation barrier and reduces the risk of breaking existing setups. The tested environment is CUDA v13.0 with Python 3.13, and the installation process has been verified on a fresh ComfyUI install.

The author also provides several example workflows covering staged generation, score editing, reloading run records, and VRAM offloading during the synthesis stage. Both plan and run outputs can be saved and reused.

VRAM and Hardware Requirements: Still Experimental

The author is very upfront about the experimental nature of this project. So far, full testing has only been done on a single RTX 5090, and the README includes measured VRAM usage data along with rough estimates for lower-VRAM cards.

For users with limited VRAM, the project includes an experimental VRAM offloading mechanism (demonstrated in workflow 06) that can significantly reduce memory usage — though the author notes that memory spikes will still occur during the process. He is also investigating optional quantization to further reduce VRAM usage, but acknowledges that the trade-off between memory savings and audio quality — particularly during the synthesis stage — still needs testing.

The author is especially looking for feedback from users with 16GB / 24GB GPUs and various ComfyUI configurations. He also notes that the generation and score inspector features have been manually tested on Nodes 2.0 and are generally functional, with some minor visual differences.

Model Weights Must Be Downloaded Separately

Note that the project does not include model weights. The README provides detailed instructions on which YuE2 files are needed and where they should be placed — users will need to obtain these themselves.

VRAM Offloading refers to moving weights or intermediate activations that aren't currently needed for computation from GPU memory (VRAM) to system RAM or disk during model inference, enabling large models to run on lower-VRAM hardware. The trade-off is that weights must be moved back to the GPU each time they're needed, introducing data transfer latency and significantly slowing overall inference speed. Model quantization compresses model weights from FP32 or FP16 precision to INT8, INT4, or lower-bit representations, reducing VRAM usage by 2–4× with minimal (or acceptable) quality loss. For audio generation models, the synthesis stage is typically most sensitive to quantization error, as small numerical deviations accumulate and affect timbral fidelity — which is precisely why the author specifically called out this stage as needing further evaluation.

The Open-Source Community's Deep Exploration of AI Music Tools

ComfyUI-Olm-YuE2 represents a valuable direction in the open-source community's exploration of AI music generation tools: rather than settling for a functional black box, it tears open every stage of the generation pipeline — exposing, decomposing, and making each step editable. For researchers and advanced creators, the value of this transparency and controllability speaks for itself.

The project is still in early experimental stages, and the author openly welcomes bug reports, edge cases, and all kinds of feedback. Interested developers can head to the GitHub repository to explore further.