STARFlow2: How Normalizing Flows Unify Multimodal Text-Image Generation

The Structural Dilemma of Unified Multimodal Models

Building unified multimodal models capable of simultaneously understanding, reasoning about, and generating interleaved text-image sequences remains a formidable challenge for both academia and industry. Existing mainstream approaches suffer from structural contradictions that are difficult to reconcile, meaning that so-called "unification" often remains superficial rather than representing true architectural fusion.

Specifically, the problems cluster around three areas:

• The fidelity bottleneck of discrete visual tokens: Approaches using discrete tokenization align conveniently with language models but sacrifice visual fidelity during image generation, resulting in noticeably degraded output quality. These methods typically rely on quantized encoders such as VQ-VAE (Vector Quantized Variational Autoencoder) or VQ-GAN, mapping continuous image features to a finite-sized codebook. At its core, this approximates a continuous visual distribution with a discrete codebook, inevitably introducing quantization error—the smaller the codebook capacity, the more severe the information loss; and even expanding the codebook brings escalating computational costs and codebook collapse issues. In contrast, continuous representation approaches operate directly in high-dimensional continuous space, preserving richer texture details and color transition information.

• Structural asymmetry between causal text generation and diffusion denoising: Stitching together autoregressive text generation with iterative diffusion-based denoising introduces a fundamental asymmetry in computational paradigms—text follows an autoregressive path while images follow a diffusion path, and the two are inherently incompatible. The core idea of Diffusion Models is to gradually add Gaussian noise to data through a forward process, then train a denoising network to progressively recover the original signal in the reverse process. This iterative denoising typically requires tens to thousands of inference steps, with each step being a global operation over the entire image. Autoregressive language models, on the other hand, generate tokens sequentially from left to right, predicting only the next element in the sequence at each step. The two paradigms differ fundamentally in temporal granularity, conditional dependency direction, and computational graph structure. Directly combining them leads to fragmented inference pipelines and makes it difficult to achieve unified KV-cache and attention optimizations.

• Capability degradation when retrofitting vision-language models: When researchers attempt to endow existing VLMs with generation capabilities, they often damage the understanding abilities accumulated during pretraining, resulting in a "robbing Peter to pay Paul" degradation phenomenon. Vision-Language Models (VLMs) learn rich cross-modal alignment knowledge and language reasoning capabilities from massive image-text pairs during pretraining. When fine-tuning or extending these models for image generation, gradient updates from the new task tend to disrupt the distribution of existing parameters—a phenomenon known as "catastrophic forgetting" in transfer learning. This is especially severe when generation tasks introduce entirely new decoding pathways or loss functions, making original understanding capabilities more susceptible to being overwritten. Common mitigation strategies include parameter freezing, low-rank adaptation methods like LoRA, and progressive training strategies, but these typically offer only partial relief and cannot fundamentally resolve conflicts arising from heterogeneous architectures.

The Fundamental Isomorphism Between Normalizing Flows and Autoregressive Transformers

The core insight of STARFlow2 lies in a key observation: autoregressive normalizing flows are essentially autoregressive Transformers.

Before diving deeper into this claim, it helps to review the mathematical principles behind normalizing flows. Normalizing Flows are a class of generative models based on invertible transformations. The core idea is to map a simple base distribution (such as a standard Gaussian) into a complex target data distribution through a series of invertible and differentiable transformations. Because each transformation step is invertible, the model can leverage the change of variables formula to compute exact log-likelihoods of the data, enabling precise density estimation rather than approximate inference. This property distinguishes normalizing flows from VAE's variational lower bound and GAN's adversarial training. In recent years, with the development of architectures such as coupling layers, autoregressive flows, and continuous normalizing flows, normalizing flows have demonstrated increasingly strong performance in image generation, speech synthesis, and other domains.

This claim is not a superficial analogy but is grounded in deep consistency at the mechanism level. Normalizing flows and large language models share the same causal mask, the same KV-cache mechanism, and the same left-to-right sequential generation structure. The causal mask is a core mechanism in the Transformer decoder that ensures when predicting the t-th token, attention can only attend to positions 1 through t-1, maintaining strict left-to-right generation order. KV-cache is a critical inference acceleration strategy: during token-by-token generation, previously computed Key and Value vectors are cached so that subsequent steps only need to compute the Query for the new token and perform attention with the cache, avoiding redundant forward passes and reducing inference complexity from O(n²) to O(n). STARFlow2's emphasis on the compatibility of normalizing flows with these mechanisms means that image generation can directly reuse the mature inference infrastructure of LLMs.

In other words, normalizing flows are highly aligned with the Transformer architecture powering today's LLMs in both mathematical form and computational workflow.

The implications of this isomorphism are profound—it means we don't need to introduce a separate mechanism (such as diffusion denoising) that diverges from the language modeling paradigm for image generation. Instead, we can handle text and images in a unified manner within the same autoregressive framework. Normalizing flows inherently possess the ability to model continuous probability densities, avoiding the fidelity loss from discrete tokenization; at the same time, they are fully compatible with the autoregressive structure of language models, eliminating the structural asymmetry between text and image generation pathways.

Why Normalizing Flows Are the Most Natural Bridge Between Language Models and Image Generation

From the perspective of architectural unification, normalizing flows are considered the most natural choice precisely because they require no compromise to the fundamental structure of the Transformer. They preserve continuous representations to ensure visual quality while adhering to causal autoregressive generation logic for compatibility with language modeling. This allows the same attention mechanism, the same caching strategy, and the same inference pipeline to flow seamlessly between text and image modalities.

Notably, this architectural unity also brings significant engineering advantages. The current LLM ecosystem has already built a mature optimization toolchain around the Transformer decoder—including efficient KV-cache management, tensor parallelism strategies, speculative decoding, and other inference acceleration techniques. The isomorphism between normalizing flows and Transformers means that this existing engineering infrastructure can be directly reused without developing a separate inference engine for the image generation modality, dramatically reducing the deployment complexity of unified multimodal models.

Three Key Technical Advantages of the STARFlow2 Unified Architecture

STARFlow2 is specifically designed to address the three-fold dilemma described above. By incorporating image generation into the same autoregressive normalizing flow framework used for text generation, the model achieves balance across the following dimensions.

Preserving Visual Fidelity

Normalizing flows model continuous distributions, avoiding the information loss that comes from compressing images into discrete tokens, and producing higher-quality image outputs. Specifically, normalizing flows operate directly in continuous latent space through invertible transformations, and their exact log-likelihood optimization objective ensures the model can capture subtle structures in the image distribution—from high-frequency texture details to low-frequency global tonal variations. This stands in stark contrast to the unavoidable quantization truncation in VQ-based approaches.

Achieving Structural Symmetry

Text and images no longer belong to two fundamentally different generation paradigms but are unified within the same left-to-right autoregressive process. This not only simplifies model design but also makes the generation of interleaved text–image sequences more natural and coherent. Interleaved text-image sequences refer to a generation format where text and images are naturally interspersed according to semantic logic within a continuous output. This capability is crucial for building truly practical multimodal AI: for example, automatically generating tutorial documents with illustrations, creating richly illustrated story narratives, or inserting visual content at appropriate points during a conversation based on context. Traditional approaches typically need to generate text first and then separately invoke an image generation model, lacking tight contextual association between the two stages. A unified model can seamlessly switch modalities at any point in the sequence generation, ensuring that image content is highly semantically consistent with the surrounding text.

Continuity of Pretrained Capabilities

Because normalizing flows share architectural origins with LLMs, image generation capabilities can be extended more smoothly while preserving the original language understanding and reasoning abilities, mitigating the performance degradation commonly seen when retrofitting VLMs. This smooth extension is feasible because normalizing flows don't require introducing new modules that are incompatible with the original Transformer architecture—no additional U-Net denoiser, no separate diffusion scheduler, and no alternative training objectives that conflict with autoregressive logic. The entire model can be optimized end-to-end within the same training framework, keeping perturbations to existing language knowledge in parameter space to a minimum.

Implications for the Multimodal Generation Paradigm

The approach represented by STARFlow2 points to a noteworthy new direction for the development of unified multimodal models. In recent years, the research community has extensively explored "how to unify understanding and generation," from discrete token approaches (such as the dVAE route originally adopted by DALL-E) to diffusion hybrid approaches (such as various attempts to embed Stable Diffusion modules within LLMs), each with its own trade-offs. Reexamining normalizing flows as "a special kind of autoregressive Transformer" opens a path that both adheres to the autoregressive paradigm and ensures continuous high-fidelity generation.

From a broader perspective, this research direction also reflects a deeper trend in AI—architectural convergence. From Transformers unifying encoder-decoder architectures in NLP, to ViT gradually replacing CNN as the mainstream backbone in vision, to multimodal efforts now attempting to unify text and image generation within a single autoregressive framework, we can observe a clear developmental trajectory: architectures that cover the most task types with the fewest structural variants will ultimately prevail under the pressures of efficiency and scalability.

For practitioners following cutting-edge AI research, the value of this work lies not only in specific performance improvements but also in the architectural philosophy it reveals: rather than awkwardly stitching together heterogeneous paradigms, it's better to find components that are truly isomorphic at the mechanism level and unify them. When normalizing flows and language models are shown to share the same causal structure, the long-standing "fragmentation" in multimodal generation may be approaching a genuine moment of convergence.

As this line of research continues to deepen, we have good reason to expect a more concise, unified, and high-fidelity multimodal generation framework to gradually take shape—one in which text and images within the same model are no longer two separate worlds, but a natural continuation of the same sequence.

Key Takeaways