[KongchangAI]
· 2 min read· 1,405 words

Split and Bridge: A New Paradigm for Multimodal Diffusion Generation Without Joint Training

Split and Bridge: A New Paradigm for Multimodal Diffusion Generation Without Joint Training

SnB bridges pretrained unimodal diffusion models at the sampling stage, enabling multimodal generation without joint training.

*Split and Bridge* proposes a bottom-up multimodal generation framework, SnB: instead of jointly training a unified model, it reuses pretrained unimodal diffusion models, letting each modality share a diffusion trajectory from the same noise during sampling, then branch at a split point, with a lightweight bridge module maintaining cross-modal consistency. The design is plug-and-play and scalable, achieves a strong balance between quality and consistency on benchmarks such as PolyMNIST, CelebAMask-HQ, PIE-Bench++, and ROCO, and extends zero-shot to interleaved co-generation, image editing, and style coupling/decoupling. The paper positions it as a practical complement to top-down joint training, and the code is open-sourced.

The field of multimodal generation has long been dominated by a "top-down" approach: jointly train a large model from scratch and let it learn a shared latent space on its own. This paradigm works well, but the costs are obvious—high training expenses, poor scalability, and the need to start over every time the modality combination changes. A recent work titled Split and Bridge: Multimodal Generation via Diffusion Bridging proposes another idea: since unimodal diffusion models are already powerful enough, why not "bridge" them at the sampling stage instead of training a unified model from scratch?

Split and Bridge paper source screenshot

From "Learning a Shared Space" to "Walking a Shared Trajectory"

The core insight of SnB (Split and Bridge) is that multimodal data (such as images and segmentation masks) essentially describe different aspects of the same underlying structure. Rather than making the model struggle to learn a unified latent variable, it is better to let the diffusion processes of different modalities first share a segment of trajectory, then branch off after a certain "split point" into modality-specific generation paths.

Specifically, during sampling, all modalities start from the same noise origin and advance along a shared diffusion trajectory to the split point; thereafter, a "bridge module" guides each modality to turn toward its own trajectory and continue denoising. The role of the bridge module is to maintain consistency among modalities after branching, preventing them from going their separate ways and ultimately generating mismatched results.

The cleverness of this design lies in its "bottom-up" nature: it does not alter the structure or weights of any pretrained unimodal model, and only operates at the sampling stage. The paper positions it as a practical complement to existing top-down multimodal methods, not a replacement.

Diffusion Model is a class of generative models whose core idea is to progressively corrupt data into pure noise through gradual noising, then train a network to learn the reverse denoising process, thereby gradually recovering data samples from noise. Sampling means starting from random noise and iteratively generating the target along the learned denoising trajectory. In multimodal scenarios, each modality can have an independent diffusion process; the key operation of SnB is to let different modalities share the same segment of denoising trajectory in the early sampling stage, which requires the diffusion timesteps and noise schedules of each modality to be aligned. The shared trajectory essentially provides a temporary "common coordinate system" for different modalities, giving them a certain structural correspondence before branching, thereby reducing the difficulty of subsequent bridge guidance.

Bridge Module is the key component in SnB for maintaining cross-modal consistency. When each modality enters its own dedicated denoising path after the split point, they may gradually deviate from their corresponding structural relationships; the role of the bridge module is to inject the current state information of other modalities into the generation process of the current modality at each denoising step, forming a cross-modal constraint or guidance. It is typically a lightweight network whose parameter scale is far smaller than the underlying unimodal diffusion models, so training costs are low and it can quickly adapt to new modality combinations. The design of the bridge module enables SnB to achieve inter-modal alignment solely through modular guidance at the sampling stage without modifying the weights of pretrained models.

The Engineering Value of Plug-and-Play

The most immediate appeal of SnB is its engineering lightness. Since no joint training is required, researchers can directly reuse the powerful unimodal diffusion models already trained in the community, combining them in a "plug-and-play" manner. In practice, this means several things:

  • Scalability: When adding a new modality, you only need to connect the corresponding pretrained model and a bridge module, without retraining the entire system.
  • Low adaptation cost: When switching datasets or tasks, the underlying unimodal capabilities do not need to be relearned.
  • Lower barrier to entry: Researchers without large-scale joint training resources can also conduct multimodal generation experiments.

The paper emphasizes that this reuse requires "without joint training," which is also the most essential difference between SnB and mainstream approaches.

Balancing Quality and Consistency on Benchmarks

In the experimental section, SnB validates its effectiveness on multiple standard benchmarks, including PolyMNIST and CelebAMask-HQ, as well as more complex PIE-Bench++ and ROCO. The paper's conclusion is that SnB achieves a "strong balance" between sample quality and cross-modal consistency.

It is worth noting that a classic challenge in multimodal generation is precisely the trade-off between quality and consistency—each modality may look realistic on its own, but they do not match when put together. SnB attempts to lock in this correspondence even after branching through shared trajectories plus bridge guidance. The paper does not provide specific quantitative comparison numbers in the abstract, but it reports competitive performance on multiple benchmarks of increasing complexity, covering the range from simple synthetic data all the way to real-world scenario datasets.

PolyMNIST is a classic synthetic benchmark in the multimodal generation field, where MNIST handwritten digits are transformed in different ways (such as different backgrounds and font styles) to generate multiple modality views, used to test whether a model can maintain semantic consistency while generating each modality. CelebAMask-HQ is a real face dataset that provides face images and their corresponding semantic segmentation masks, commonly used for joint generation and editing tasks involving images and masks. PIE-Bench++ and ROCO further increase complexity: the former is oriented toward image editing evaluation, examining consistency before and after editing; the latter contains paired medical images and text descriptions, involving cross-modal semantic alignment. These benchmarks form an escalating evaluation chain from synthetic to real, from simple pairing to complex semantics, comprehensively measuring SnB's generation quality and cross-modal consistency.

Natural Extension to Zero-Shot Downstream Tasks

Another highlight of SnB is its natural extension to several zero-shot downstream tasks:

Interleaved co-generation: simultaneously generating multiple corresponding modalities, such as an image and its segmentation mask.

Image editing: modifying the content of one modality while maintaining constraints from other modalities through the bridge mechanism.

Style coupling/decoupling: controlling the degree of style association between different modalities—they can be either bound together or separated.

These tasks require no additional training and directly benefit from SnB's sampling framework. This also indirectly shows that the structure of "shared trajectory + branching" inherently contains cross-modal alignment capabilities, rather than merely generating realistic samples.

Zero-shot here means that the model completes a task without additional training specific to that downstream task, relying solely on the existing generation framework and pretrained weights. The reason SnB can extend zero-shot to interleaved co-generation, image editing, and style coupling/decoupling is that its sampling framework itself encodes cross-modal correspondence in the shared trajectory and bridge guidance: changing the initial noise or intermediate state of one modality will naturally propagate the change to other modalities through the bridge module, thereby supporting conditional generation and editing. This "structure as capability" characteristic reduces dependence on task-specific fine-tuning data and lowers the barrier to deploying new tasks.

The Significance of the Bottom-Up Paradigm

The greater value of SnB may lie in the question it raises that deserves serious consideration: must multimodal generation always involve joint training from scratch? In an era when pretrained models are becoming increasingly powerful and accessible, compositional methods at the sampling stage offer a lower-cost, more flexible path. The authors explicitly position SnB as a "complement" rather than a "replacement"—top-down joint training still has advantages in scenarios requiring deep modality fusion, while SnB excels in reuse, extension, and rapid adaptation.

The code is open-sourced on GitHub (AhmadArrabi/SnB), making this a direction worth watching for researchers who want to experiment with multimodal diffusion generation but lack large-scale training resources.

One-line Summary

SnB bridges pretrained unimodal diffusion models at the sampling stage, enabling multimodal generation without joint training.

Paragraph Summary

Split and Bridge proposes a bottom-up multimodal generation framework, SnB: instead of jointly training a unified model, it reuses pretrained unimodal diffusion models, letting each modality start from the same noise and share a diffusion trajectory during sampling, then branch at a split point, with a lightweight bridge module maintaining cross-modal consistency. The design is plug-and-play, scalable, and low in adaptation cost, achieving a strong balance between quality and consistency on benchmarks such as PolyMNIST, CelebAMask-HQ, PIE-Bench++, and ROCO, and can extend zero-shot to interleaved co-generation, image editing, and style coupling/decoupling. The paper positions it as a practical complement to the top-down joint training paradigm, and the code has been open-sourced.

#multimodal generation#diffusion models#Split and Bridge#joint training#zero-shot#image editing
Share:

Related articles