Normalizing Flow

Normalizing flows can use the change-of-variables formula to exactly compute the log-likelihood of data through invertible transformations, enabling precise density estimation rather than approximate inference.

This paper's Pull-back algorithm is similar to normalizing flows in that it also adopts a layer-by-layer stacking strategy, but it does not require the transformation to be invertible, nor does it require a closed-form Jacobian determinant; each layer only needs to learn a Stein displacement field, which makes the Pull-back algorithm more flexible in transformation form than traditional normalizing flows.

Normalizing flows achieve exact density estimation through invertible transformations, and the target density is computed by the change-of-variables formula p_K(x) = p_0(f^{-1}(x)) ∏|det ∇f_k^{-1}(x)|; their advantage is that the log-likelihood can be computed exactly, but the cost is that each layer's transformation must satisfy the two constraints of invertibility and an easily computable Jacobian determinant, which severely limits model expressiveness.

This framework introduces normalizing flows on the conditional distributions to capture highly non-Gaussian, nonlinear posterior shapes.

An arXiv paper (arXiv:2610.10870v1) proposes solving the Poisson equation by learning a variable transformation that maps complex targets back to a reference space where the equation can be solved analytically.

The paper trains a bijection (typically implemented as a normalizing flow) so that the target distribution in the latent space approximates a reference density with simple structure.

Importance sampling (IS) from a normalizing flow is exactly the limiting case of this control variate construction, where the control variate method converges to the importance sampling form when the latent-space sampler degenerates to i.i.d. sampling.

Normalizing flows are a class of deep generative models that construct complex probability distributions by stacking invertible, differentiable transformations, and express densities using the change-of-variables formula combined with the Jacobian determinant.

In this paper's setting, normalizing flows are not used to generate new samples, but rather to provide a coordinate transformation that flattens the complex target density into a reference density.

The training overhead of the normalizing flow itself and whether the tail conditions hold for real-world targets are factors that need to be weighed when deploying the method in practice.

The core contribution of this paper lies in bridging deep generative models such as normalizing flows with classical variance reduction theory.

Normalizing flows stand in contrast to VAEs (which require variational lower-bound approximation) and GANs (which cannot compute likelihoods)

To guarantee invertibility, normalizing flows impose strong constraints on network architecture (such as RealNVP's affine coupling layers and Glow's 1×1 invertible convolutions), which limits the expressive capacity of individual transformations and increases parameter count and computational cost

Normalizing flows can directly reuse the mature inference infrastructure of LLMs, including KV-cache management, tensor parallelism strategies, and inference acceleration techniques such as speculative decoding.

Normalizing flows exactly compute the log-likelihood of data via the change-of-variables formula: log p(x) = log p(f(x)) + log |det(∂f/∂x)|.

Normalizing flows model continuous distributions, avoiding the information loss caused by compressing images into discrete tokens.

Autoregressive normalizing flows are essentially autoregressive Transformers.

STARFlow2 achieves visual fidelity preservation, structural symmetry, and continuity of pretrained capabilities by incorporating image generation into an autoregressive normalizing flow framework consistent with text generation.

Normalizing flows share with large language models the same causal mask, KV-cache mechanism, and a consistent left-to-right sequence generation structure.

A unified autoregressive framework allows the optimization toolchain from the LLM ecosystem—including KV-cache management, tensor parallelism strategies, and speculative decoding—to be directly reused for image generation.