UniAVC
A unified audio-visual generation model that adopts a progressive training strategy and a modality-decoupled scheduler, supporting multiple tasks such as joint generation, cross-modal editing, and self-refinement.
Timeline (last 90 days)
UniAVC proposes a unified audio-video generation paradigm that supports text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing.
UniAVC adopts a progressive training strategy: first each branch is trained on single-modality datasets for generation and editing capabilities, then paired data is used to mine cross-modal correspondences.
One of UniAVC's core innovations is a modality-decoupled scheduler that, starting from a shared timestep, warps the noise schedule independently for each modality.
During training, UniAVC directly supervises asymmetric configurations with only one clean modality, enabling the model to learn to preserve one modality and constrain the generation of the other.
UniAVC covers multiple tasks such as joint generation, cross-modal editing, and self-refinement with a single set of weights, with no need to attach a separate adapter for each task.
The effectiveness of the UniAVC scheme depends on the quality and scale of the paired data, as well as the tuning robustness of the decoupled scheduler under different modality combinations.
All Facts (6)
UniAVC proposes a unified audio-video generation paradigm that supports text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing.
50%UnverifiedUniAVC adopts a progressive training strategy: first each branch is trained on single-modality datasets for generation and editing capabilities, then paired data is used to mine cross-modal correspondences.
50%UnverifiedOne of UniAVC's core innovations is a modality-decoupled scheduler that, starting from a shared timestep, warps the noise schedule independently for each modality.
50%UnverifiedDuring training, UniAVC directly supervises asymmetric configurations with only one clean modality, enabling the model to learn to preserve one modality and constrain the generation of the other.
50%UnverifiedUniAVC covers multiple tasks such as joint generation, cross-modal editing, and self-refinement with a single set of weights, with no need to attach a separate adapter for each task.
50%UnverifiedThe effectiveness of the UniAVC scheme depends on the quality and scale of the paired data, as well as the tuning robustness of the decoupled scheduler under different modality combinations.
50%