InstantFusion
A research framework that proposes a lightweight shared latent-space interface enabling interoperability and collaboration between heterogeneous image generation models, specifically diffusion models and flow-matching models. It employs model-specific Latent Autoencoder (LAE) modules together with dual alignment constraints to achieve bidirectional mapping of denoising trajectories across architecturally distinct generative models.
Timeline (last 90 days)
InstantFusion research found that even independently trained, unrelated generators exhibit similar latent space organizational structures.
Directly handing off the intermediate denoising state of one model to another unaligned model still preserves the rough visual content of the image.
InstantFusion is a lightweight shared latent space interface for connecting the denoising trajectories of heterogeneous image generators.
InstantFusion introduces a model-specific latent autoencoder (LAE) for each generator, projecting intermediate states into a shared latent space and decoding them back into the latent space format required by the target generator.
InstantFusion constrains the shared latent space through two complementary training objectives: state alignment and velocity synchronization.
Aligning only the intermediate state at a certain moment is insufficient to ensure consistency in the direction of the denoising trajectory after handoff; the velocity synchronization constraint is equivalent to simultaneously aligning both the position and tangent direction of the trajectory.
After training, InstantFusion requires no changes to any original generator weights, and all collaborating models remain frozen.
The paper validates InstantFusion's generality across four types of tasks—acceleration, multi-preference composition, cross-architecture online policy distillation, and cross-model composition—achieving competitive performance.
InstantFusion's shared interface makes cross-architecture online policy distillation possible, allowing a teacher model (e.g., a large DiT) and a student model (e.g., a lightweight UNet) to exchange intermediate states in a shared latent space.
The results presented in the paper are from the research stage, and their robustness under large-scale models and complex prompts remains to be verified by subsequent work.
All Facts (10)
InstantFusion research found that even independently trained, unrelated generators exhibit similar latent space organizational structures.
50%UnverifiedDirectly handing off the intermediate denoising state of one model to another unaligned model still preserves the rough visual content of the image.
50%UnverifiedInstantFusion is a lightweight shared latent space interface for connecting the denoising trajectories of heterogeneous image generators.
50%UnverifiedInstantFusion introduces a model-specific latent autoencoder (LAE) for each generator, projecting intermediate states into a shared latent space and decoding them back into the latent space format required by the target generator.
50%UnverifiedInstantFusion constrains the shared latent space through two complementary training objectives: state alignment and velocity synchronization.
50%UnverifiedAligning only the intermediate state at a certain moment is insufficient to ensure consistency in the direction of the denoising trajectory after handoff; the velocity synchronization constraint is equivalent to simultaneously aligning both the position and tangent direction of the trajectory.
50%UnverifiedAfter training, InstantFusion requires no changes to any original generator weights, and all collaborating models remain frozen.
50%UnverifiedThe paper validates InstantFusion's generality across four types of tasks—acceleration, multi-preference composition, cross-architecture online policy distillation, and cross-model composition—achieving competitive performance.
50%UnverifiedInstantFusion's shared interface makes cross-architecture online policy distillation possible, allowing a teacher model (e.g., a large DiT) and a student model (e.g., a lightweight UNet) to exchange intermediate states in a shared latent space.
50%UnverifiedThe results presented in the paper are from the research stage, and their robustness under large-scale models and complex prompts remains to be verified by subsequent work.
50%