UniAVC: A Unified Audio-Video Generation Paradigm with Asynchronous Scheduling to Solve Cross-Modal Challenges

The Technical Dilemma of Joint Audio-Video Generation

Unified audio-video creation has always been a thorny problem: getting a single model to simultaneously handle text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing sounds great in theory but is extremely difficult in practice. Existing methods rely heavily on paired corpora, which are inherently scarce in joint generation scenarios and even rarer when editing tasks are involved. As a result, models spend most of their capacity learning single-modality generation and editing capabilities that can be taught with single-modality data, while neglecting the cross-modal coupling relationships that genuinely require paired data to model.

More critically, current systems adopt a unified noise scheduling strategy for both audio and video, which introduces two overlooked flaws. First, video has a much higher information density than audio and requires a longer high-noise phase to organize spatial and motion structure, whereas audio signals are relatively sparse and could stabilize earlier; a unified schedule may harm this dynamic difference. Second, synchronized denoising leaves asymmetric workflows—such as conditioning on one modality to synthesize or edit the other—completely untrained, making the model unable to handle such practical needs.

Diffusion Models and Noise Scheduling: Most mainstream audio-video generation models today are based on diffusion models, whose core idea is to start from pure noise and gradually denoise to recover the target data. The noise schedule determines the noise intensity at each time step, directly affecting the denoising rhythm. Video, containing spatial structure and motion information, has high information density and typically needs to stay longer in the high-noise phase to organize global structure; audio signals are relatively sparse with strong temporal continuity and can stabilize earlier in the low-noise phase. A unified schedule means the two modalities are forced to share the same denoising rhythm, making it difficult to accommodate their respective information characteristics.

Paired Corpora and Asymmetric Workflows: Paired corpora refer to audio-video data that is paired and semantically aligned, which is key to learning cross-modal coupling relationships. Asymmetric workflows refer to tasks that condition on one modality to generate or edit another—for example, generating matching audio given video, or editing video content given audio. If such tasks are not explicitly covered during training, the model cannot handle them at inference time.

UniAVC's Progressive Training Strategy

To address the pain point of scarce paired data, UniAVC proposes a staged training scheme. It first trains each branch on single-modality datasets until it has complete generation and editing capabilities, fully leveraging abundant single-modality data. Only then is the precious paired data dedicated to mining cross-modal correspondences. This "single first, dual later" approach concentrates the value of paired data on cross-modal coupling rather than wasting it on single-branch tasks.

This progressive training also brings additional benefits: the model establishes a solid generation foundation in the single-modality stage, so subsequent cross-modal training does not need to learn representations from scratch. Compared with end-to-end direct training, this strategy is more data-efficient and easier to extend to new tasks.

Modality-Decoupled Scheduler and Asymmetric Configurations

One of UniAVC's core innovations is the modality-decoupled scheduler. Starting from shared time steps, it independently warps the noise schedule for each modality, giving video and audio their own denoising rhythms suited to their respective information densities. This way, video can stay longer in the high-noise phase to build structure, while audio can converge earlier.

Another key design is direct supervision of asymmetric configurations. For subtasks with only one clean modality, training directly lets such configurations participate in supervision, so the model learns to work in a "one moving, one still" manner—keeping one modality intact while using it to constrain the generation of the other. This unlocks two practical capabilities: intent-aware editing, i.e., editing one modality while keeping another unchanged; and post-inference self-refinement, i.e., for an accepted branch, regenerating only the unsatisfactory other branch without redoing everything.

Schedule Warping: UniAVC's modality-decoupled scheduler does not simply set different total step counts for the two modalities; rather, starting from a shared time-step axis, it applies independent time-warping functions to each modality, mapping uniform time steps onto their respective noise-level curves. This way, video and audio can undergo different noise-intensity trajectories under the same number of sampling steps, achieving a "same clock, different rhythm" denoising process. The key to this design is that the warping functions must remain monotonic to ensure the denoising process is invertible and stable.

Self-Refinement and Intent-Aware Editing: Self-refinement refers to, after joint generation, keeping an accepted modality unchanged and only resampling and denoising the other unsatisfactory modality, thereby improving overall quality at lower cost. Intent-aware editing requires the model to understand the intent of "keeping one modality unchanged while modifying the other according to instructions," which relies on direct supervision of asymmetric configurations during training, enabling the model to learn to generate the target modality under conditional constraints.

A Single Set of Weights Covers Multiple Tasks

Thanks to the above designs, UniAVC can cover multiple creation tasks with a single set of weights, without needing a separate adapter for each task. From joint generation to cross-modal editing to self-refinement, the model switches flexibly within a unified framework. This is significant for practical deployment—reducing the number of models, lowering maintenance costs, and enabling knowledge sharing across tasks.

Of course, the effectiveness of this scheme still depends on the quality and scale of paired data, as well as the tuning robustness of the decoupled scheduler under different modality combinations. But in terms of approach, UniAVC makes cross-modal coupling the focus of training rather than letting the model over-consume capacity on single-branch tasks—this shift in perspective is worth noting.

One-line Summary

UniAVC uses progressive training and a modality-decoupled scheduler to crack the challenges of scarce paired data and unified scheduling in joint audio-video generation.

Paragraph Summary

Existing joint audio-video generation methods rely heavily on scarce paired corpora and adopt unified noise scheduling for audio and video, ignoring the information-density differences between them and leaving asymmetric workflows untrained. UniAVC proposes a progressive training strategy: first train each branch on abundant single-modality data to have complete generation and editing capabilities, then dedicate paired data specifically to learning cross-modal coupling. Its core innovations include a modality-decoupled scheduler that independently warps the noise rhythm for each modality, and direct supervision of asymmetric configurations that unlocks intent-aware editing and post-inference self-refinement. Ultimately, a single set of weights covers multiple tasks without task-specific adapters, offering advantages in data efficiency and deployment convenience, though effectiveness remains constrained by paired data quality and scheduler tuning robustness.