InstantFusion: A Shared Latent Space Interface for Heterogeneous Image Generation Models

When There Are More and More Generative Models, How Can They Collaborate?

Diffusion models and flow matching models have become the de facto standards in image generation, giving rise to a large number of generators with vastly different architectures and representations. From the early UNet structure to the now-popular DiT (Diffusion Transformer), different models exhibit significant differences in their latent space organization and denoising dynamics. This heterogeneity raises a long-overlooked question: Can the latent representations of different models be aligned? Do their respective denoising trajectories share any intrinsic consistency?

InstantFusion tackles this question head-on. The researchers discovered an intriguing phenomenon — even generators that were trained independently with no connection to each other exhibit similar latent space organizational structures. Moreover, directly handing off the intermediate denoising state of one model to another unaligned model still manages to preserve the rough visual content of the image. This cross-model compatibility provides a theoretical foundation for inter-model collaboration.

The architectural differences between diffusion models and flow matching models are key context for understanding the challenges of cross-model collaboration. Diffusion models generate images by progressively adding Gaussian noise and learning the reverse denoising process; flow matching models learn a continuous probability flow (an ODE trajectory) from the noise distribution to the data distribution, typically offering more stable training and fewer inference steps. Architecturally, early diffusion models relied on UNet (a convolutional encoder-decoder structure), while DiT replaces convolutional layers with Transformer blocks, achieving superior performance at high resolutions. The two model families differ not only in their backbone networks, but also in their latent space dimensionality, encoding conventions, and noise schedules. This is precisely why directly "bridging" their intermediate denoising states seemed intuitively impossible — yet InstantFusion found that an implicit compatibility actually exists.

The Core Design of InstantFusion

Building on these observations, the research team proposed InstantFusion — a lightweight shared latent space interface for connecting the denoising trajectories of heterogeneous image generators. Its goal is not to replace any existing model, but to act as a "translation hub" that allows different models to seamlessly hand off the generation process to one another.

Model-Specific Latent AutoEncoders (LAE)

The key to enabling cross-model state transfer is introducing a model-specific Latent AutoEncoder (LAE) for each generator. These LAEs serve two purposes: first, projecting intermediate states from different generators into a shared latent space; second, decoding states from the shared space back into the latent format required by the target generator.

The elegance of this design lies in how it avoids the combinatorial explosion problem of the traditional approach, where a dedicated mapping must be trained for every pair of models. As long as each model is connected to the shared latent space, any two models can interoperate freely, dramatically reducing the cost of scaling.

Traditional cross-model knowledge transfer typically relies on Knowledge Distillation or model alignment methods, but these approaches suffer from a "combinatorial explosion" problem: with N models, N×(N-1) dedicated mapping networks must be trained for every pair, making scalability extremely poor. The LAE design draws on the "intermediate representation bottleneck" idea — similar to how multilingual translation in NLP uses a pivot language as a hub: each model only needs to interface with the shared space, reducing the total number of mappings to 2N. The autoencoders themselves are typically lightweight networks (such as small MLPs or shallow Transformers) that don't require training the generators from scratch, making the training cost far lower than full model fine-tuning. This "star topology" interface design means that adding a new generator only requires training its corresponding LAE, without touching existing models or retraining other LAEs.

Two Complementary Training Objectives

The shared latent space is not built arbitrarily — it is shaped by two complementary training objectives:

• State Alignment: Aligning intermediate states derived from the same image at the same noise level within the shared space;
• Velocity Synchronization: Synchronizing their corresponding local denoising velocities.

The former ensures that "the same content occupies a consistent position across different models," while the latter ensures "continuity in denoising dynamics." Only by combining both can the denoising process switch between different models without content collapse or trajectory discontinuity.

Denoising Velocity is a central concept in the flow matching framework, referring to the direction vector of how the latent variable changes over time at a given timestep along the denoising ODE trajectory (i.e., the velocity field $v_\theta(x_t, t)$). In diffusion models, the analogous concept is the predicted noise or score function. The significance of synchronizing denoising velocities is this: aligning intermediate states at a single moment (positional alignment) is not sufficient to guarantee that the denoising trajectory directions will be consistent after a handoff — two models may be at the same position yet denoising in different directions, causing the generated content to diverge after the handoff. Velocity synchronization effectively aligns both the "position" and the "tangent direction" of the trajectory, allowing the denoising process after a model switch to evolve along an approximately consistent manifold, preventing abrupt content changes or quality collapse.

Plug-and-Play with Frozen Models

An important engineering advantage of InstantFusion is that no original generator weights need to be modified after training. All participating models remain frozen, and the denoising process can transition freely among them without requiring dedicated mappings for each model pair.

In practice, this means developers can directly "plug" their existing, independently trained generative models into this shared interface without retraining or fine-tuning the backbone networks. For resource-constrained scenarios, this decoupled design significantly lowers the barrier to entry.

Four Application Scenarios

The paper demonstrates InstantFusion's versatility across four distinct tasks:

Acceleration

By leveraging each model's strengths at different stages of the denoising trajectory, relay-style generation across models achieves acceleration, striking a better balance between speed and quality.

Multi-preference Composition

Different models are often trained for different aesthetic preferences or data distributions. Through the shared interface, the preferences of multiple models can be fused in a single generation pass, producing results that combine stylistic characteristics from multiple sources.

Cross-architecture On-policy Distillation

Breaking through the limitations of homogeneous distillation, knowledge can now be transferred between models of different architectures. This has practical significance for compressing large model capabilities into lightweight models.

On-policy Distillation differs from offline distillation in its training data source: offline distillation uses a static dataset pre-generated by the teacher model, while on-policy distillation generates samples in real time using the current student model during training and uses the teacher model's feedback as the supervision signal, thereby avoiding the distribution shift problem. Traditional on-policy distillation requires the teacher and student to share the same latent space or at least the same architecture (e.g., both being DiT). InstantFusion's shared interface makes cross-architecture on-policy distillation possible: the teacher model (e.g., a large DiT) and the student model (e.g., a lightweight UNet) can exchange intermediate states in the shared latent space, allowing the teacher's denoising trajectory to directly guide the student's learning, thereby compressing large model generation capabilities into smaller models of different architectures — balancing deployment efficiency and generation quality.

Cross-model Composition

Combining multiple heterogeneous generators into a collaborative system that plays to each model's strengths. Research shows that InstantFusion achieves competitive performance across all of the above scenarios.

Significance and Outlook

The value of InstantFusion lies not only in the specific method itself, but also in how it reveals and exploits the latent representational consistency among heterogeneous generative models. As the image generation model ecosystem becomes increasingly fragmented — with every team and every product having its own model — a "universal interface" that allows these models to interconnect becomes ever more important.

In the long run, this kind of shared latent space approach could extend to video, audio, and even multimodal generation, providing a new paradigm for building composable, collaborative generative AI infrastructure. Of course, what is presented in the paper remains research-stage work, and its robustness under large-scale models and complex prompts still requires further validation in future work.