Annealed Lévy Diffusion: A New Paradigm for Heavy-Tailed Diffusion via Gaussian Surrogates

Diffusion models have become a mainstream framework for generative modeling, but standard Gaussian diffusion has inherent limitations when capturing heavy-tailed distributions. Denoising Lévy Probabilistic Models (DLPM) address this issue by introducing heavy-tailed perturbations, yet they use a fixed tail index throughout the entire denoising process, lacking flexibility. A new paper proposes Annealed Lévy Diffusion (ALD), which attempts to find a balance between heavy-tailed exploration and Gaussian refinement.

Core Idea: Making the Stability Index Time-Varying

The core innovation of ALD is turning the stability index $\alpha(t)$ from a fixed value into an annealing function that varies with time $t$. In the early stages of the diffusion process, larger heavy-tailed characteristics provide stronger exploration capability; as the process progresses, $\alpha(t)$ gradually increases, and the noise distribution approaches Gaussian, thereby achieving fine-grained sample refinement.

This design responds to a fundamental contradiction in diffusion models: heavy-tailed noise helps escape local modes and cover broader distribution regions, but Gaussian noise performs more stably in high-precision reconstruction during the reverse process. ALD uses an annealing strategy to chain the advantages of both, rather than choosing one over the other.

Stability Index and Lévy Processes: The stability index (usually denoted as α or β) is the shape parameter of the stable distribution (Lévy α-stable distribution), with a range of (0, 2]. It directly controls the tail decay rate of the distribution: the smaller the α, the heavier the tail, and the higher the probability of extreme values appearing; the variance or even the mean may not exist. When α = 2, the stable distribution degenerates into a Gaussian distribution, and the tail decays exponentially fast. Therefore, α can be seen as a knob for continuous transition from "heavy-tailed exploration" to "Gaussian refinement." In the context of diffusion models, the noising distribution is essentially a stochastic process driven by a Lévy stable process, with different α values corresponding to different intensities of jumps. Early works such as DLPM fix α throughout the entire forward and reverse processes, meaning the noise characteristics remain consistent from start to finish; whereas ALD lets α(t) evolve over time, which is equivalent to giving the forward process itself a non-stationary structure that "gradually fades from heavy-tailed to Gaussian"—this is precisely the source of its flexibility.

Marginal-First Decomposition: Expressing Stable Noise as a Gaussian Scale Mixture

The key to making ALD truly trainable is the "marginal-first" formulation proposed in the paper. The authors express stable noise as a scalar Gaussian scale mixture, which means each noising step can be characterized by a Gaussian variable multiplied by a scale factor.

The direct benefit of this transformation is that the training objective returns to the standard Gaussian noise prediction loss, requiring only an additional scalar scale input in the network input. In other words, researchers do not need to redesign the training framework for Lévy diffusion; the existing Gaussian diffusion training pipeline can be reused with minimal modifications. This greatly lowers the barrier to adopting heavy-tailed diffusion.

Gaussian Scale Mixture and Marginal-First Training: A Gaussian scale mixture means that a random variable can be represented as a standard Gaussian variable multiplied by an independent non-negative scale random variable, i.e., X = S · Z, where Z ~ N(0, I) and S is a scalar scale factor. Many heavy-tailed distributions (including stable distributions, Student's t-distribution, etc.) can be expressed in this form. "Marginal-first" refers to first deriving the marginal distribution of the noising process at any time step, and then constructing the training objective based on it, rather than directly simulating the trajectory of the entire stochastic differential equation. The advantage of this approach is that, given a clean sample x₀ and scale S, the conditional distribution of the noised sample x_t is still Gaussian. Therefore, we can train the network using denoising score matching or noise prediction loss just like standard diffusion models, only requiring the network to additionally receive scale S as an input condition. This decomposition completely brings the training problem of heavy-tailed diffusion back into the existing Gaussian diffusion framework, and is the theoretical basis for ALD's "plug-and-play" capability.

Two Samplers, One Predictor

The ALD paper studies two sampling processes that share the same trained noise predictor and differ only through different scalar inputs:

ALD-renoise adopts the strategy of "estimating the clean sample and then re-injecting marginal noise." At each step, it first denoises to obtain an estimate of $x_0$, and then samples new noise from the current marginal distribution. This approach is simple to implement, but introduces random resets between steps, which may lose the dependency structure between adjacent states.

ALD-posterior, on the other hand, maintains the dependency relationship between adjacent states through accumulated conditional variance and Gaussian bridge updates. Compared with renoise, it is closer to the Markov chain structure of traditional diffusion sampling, and theoretically provides a smoother sampling trajectory. The two samplers each have advantages on different tasks, and the paper demonstrates this comparison through experiments.

Mechanistic Differences Between ALD-renoise and ALD-posterior: ALD-renoise first estimates the clean sample x̂₀ from the current state at each step, and then resamples noise from the current marginal distribution to obtain the next state. This "estimate-then-renoise" paradigm is similar to ancestral sampling in diffusion models, but because noise is redrawn at each step, adjacent states no longer form a strict Markov chain, and the sample trajectory may appear more "jumpy." ALD-posterior adopts a posterior sampling approach: it accumulates conditional variance and uses a Gaussian bridge to construct the transition from the current state to the next state, maintaining an explicit dependency relationship between adjacent states. A Gaussian bridge is a conditional Gaussian process that interpolates the intermediate path given both endpoints, ensuring the smoothness and Markov property of the transition. Intuitively, renoise is more suitable for scenarios requiring strong noise injection to explore multimodal distributions, while posterior is more suitable for scenarios requiring high-quality, structurally coherent samples. Sharing the same predictor means that at deployment time, the sampler can be flexibly switched according to task requirements without retraining the model.

Experimental Coverage: From 2D Benchmarks to Image Generation

The paper evaluates ALD on four 2D benchmark datasets and an imbalanced nine-peak mixture distribution, while also examining training efficiency and the impact of predictive perturbations on clipped Holtsmark data.

In terms of image generation, ALD's generation quality is validated on MNIST and CelebA64, and it is used for data augmentation on CIFAR-10LT. The paper claims that ALD can preserve local geometric properties on the evaluated light-tailed targets and manifold-structured targets—this is particularly critical for data augmentation scenarios, because if augmented samples deviate from the original manifold, they may instead harm downstream model performance.

Why It Is Worth Paying Attention To

The value of ALD is not just that it is another diffusion variant. It provides a modular idea: any framework using Gaussian diffusion can gain heavy-tailed exploration capability by adding a scale input, without changing the training objective or network architecture. For researchers working on multimodal distributions, long-tailed data, or imbalanced datasets, this "plug-and-play" heavy-tailed extension has practical appeal.

Of course, the scheduling design of the hyperparameter $\alpha(t)$ in the annealing strategy, the performance of different samplers on large-scale high-resolution tasks, and the impact of heavy-tailed noise on training stability remain issues that need further verification. The paper currently focuses mainly on small- and medium-scale benchmarks, and its scalability to larger-scale generative tasks remains to be tested by future work.

One-line Summary

ALD anneals the stability index of Lévy diffusion over time, balancing heavy-tailed exploration and Gaussian refinement, while reusing the Gaussian diffusion training pipeline.

Paragraph Summary

Annealed Lévy Diffusion (ALD) addresses the lack of flexibility in Denoising Lévy Probabilistic Models caused by a fixed tail index by proposing to turn the stability index α(t) into a time-varying annealing function: in the early diffusion stage, heavy-tailed noise with smaller α enhances exploration, and later α gradually increases toward Gaussian to enable fine reconstruction. To make the model trainable, the paper adopts a "marginal-first" formulation, writing stable noise as a scalar Gaussian scale mixture, thereby returning the training objective to the standard Gaussian noise prediction loss, with only an additional scalar scale added to the network input. ALD studies two samplers, ALD-renoise and ALD-posterior, which share the same predictor; the former re-injects marginal noise at each step, while the latter uses a Gaussian bridge to maintain state dependencies. Experiments cover multiple 2D benchmarks, a nine-peak mixture, MNIST, CelebA64, and CIFAR-10LT data augmentation, validating the method's effectiveness in preserving local geometric properties.