Why Long Skip Connections Determine Feature Quality in DiT Models

The DiT Paradox: Better Generation, Weaker Representations

Features extracted from the intermediate activations of diffusion models — known as diffusion features — are becoming an increasingly important bridge between unified image generation and visual understanding. As diffusion backbones evolved from traditional U-Net architectures to Diffusion Transformers (DiT), the field gained substantially stronger generative capabilities. Yet a counterintuitive phenomenon emerged: DiT produces higher-quality images but yields weaker extracted features.

This contradiction poses a direct challenge to the vision of unified generation and understanding. If a model can render more photorealistic images yet fail to provide effective representations for downstream discriminative tasks, its value as a general-purpose visual backbone becomes questionable. This research addresses precisely that problem — and delivers a clear, verifiable answer.

The standard approach to extracting diffusion features involves running a few denoising steps on a noisy input image during inference, then capturing the intermediate layer activations of the backbone network as visual representations for downstream tasks like classification, segmentation, and depth estimation. The appeal of this approach is that diffusion models, trained on large-scale image datasets, naturally develop rich semantic awareness — in theory enabling strong features without additional discriminative pretraining. Diffusion Transformer (DiT) replaces the convolution-based U-Net backbone with standard Transformer blocks, offering better scalability and higher generation quality, as seen in Stability AI's DiT series, Flux, SD3, and related models. Yet this architectural shift inadvertently degraded discriminative feature quality, creating a growing performance gap between generation and understanding.

The Key Variable: Long Skip Connections (LSC)

Through controlled from-scratch training experiments, the researchers isolated the core factor responsible for feature quality: Long Skip Connections (LSC). The conclusion is straightforward — DiT models with LSC produce significantly better features.

The methodology here deserves emphasis. Rather than comparing off-the-shelf models with different architectures — which introduces confounds from data, scale, and training strategies — controlled from-scratch training isolates LSC as a single variable. This clean experimental design is what establishes a causal relationship between LSC and feature quality, rather than mere correlation.

Unpacking the Mechanism: How Diffusion Noise "Contaminates" Image Signals

The most valuable contribution of this paper is its mechanistic diagnosis, not just a description of the phenomenon.

The Dual Identity of the Input Signal

The input to a diffusion backbone simultaneously carries two types of information: the latent image and the diffusion noise. The key insight is that while this "noise" appears stochastic in form, it actually encodes information about the diffusion trajectory. To make a final prediction of the noisy variable — whether predicting $v$ or $\epsilon$ — the backbone must leverage both the image signal and the trajectory signal.

Why the Absence of LSC Degrades Features

This is where the problem lies. When both signals must travel through the same main trunk of the network, the noise signal "contaminates" the image signal, degrading the quality of image representations carried through the backbone. In other words, the network is forced to simultaneously preserve clean semantic information and transport trajectory-related noise signals through a single shared pathway — and the two interfere with each other.

Long skip connections provide a shortcut: they route the diffusion noise directly to the locations where it is actually needed, dramatically reducing the residual noise signal passing through the main trunk. With the trunk "purified," image semantics are preserved more cleanly, and feature quality improves accordingly.

The training objective of diffusion models is typically to predict either the noise $\epsilon$ (noise prediction) or the velocity field $v$ (v-prediction). Given a timestep $t$, the noisy input $x_t$ can be expressed as $x_t = \alpha_t x_0 + \sigma_t \epsilon$, where $\alpha_t$ and $\sigma_t$ are coefficients determined by the noise schedule. The backbone must simultaneously "read" two types of information from $x_t$: the semantic content of $x_0$ (image signal), and the noise component at timestep $t$ (trajectory signal). The trajectory signal is not truly random — throughout the denoising process, it carries implicit information about the current position along the diffusion trajectory, which is a necessary input for the network to complete its prediction task. This coupling of the two signals is the fundamental dilemma faced by backbones without LSC.

Beyond a Design Trick: The Deep Connection Between LSC and the Diffusion Objective

The deeper significance of this work lies in reframing the role of LSC. Long skip connections have long been viewed as an engineering convenience — useful, but lacking a principled explanation. This paper's analysis shows that LSC is not an optional add-on but a structural element intimately tied to the diffusion objective itself.

From this perspective, U-Net's strong feature quality is likely not coincidental — its inherent long-skip architecture naturally aligns with the diffusion task's need to "separate image signals from trajectory signals." When DiT pursued a pure Transformer architecture at the cost of LSC, it discarded this critical signal-decoupling mechanism.

U-Net's long skip connections directly concatenate encoder feature maps to their symmetric decoder layers, creating a natural signal-splitting mechanism: the encoder can focus more on extracting semantic features, while the decoder receives low-level details (and implicit noise trajectory information) via skip connections to complete pixel-level reconstruction. This structure was not designed for diffusion tasks when U-Net was originally developed (initially for medical image segmentation), yet it serendipitously aligned with the signal-decoupling requirements of the diffusion era. The purely Transformer-based DiT abandons this hierarchical encoder-decoder structure; all layers are stacked serially, forcing input signals to pass through the entire trunk, with no mechanism to separate semantic signals from trajectory signals. The contribution of this paper is to elevate this phenomenon from an empirical observation to a theoretically grounded causal explanation, providing principled guidance for future architectural design.

Implications for Unified Generation and Understanding

For researchers and engineers building unified visual backbones on top of diffusion models, this finding offers direct practical guidance:

• If you want a DiT to both generate and serve as a high-quality feature extractor, introducing long skip connections is a principled architectural choice;
• When evaluating a diffusion backbone's "understanding capability," the signal routing pathways for noise and image signals within the architecture should be considered — not just generation metrics;
• This mechanism also suggests a middle-ground approach: achieving signal decoupling through structural design while preserving the advantages of Transformers, balancing both generation and discrimination.

The researchers have open-sourced the relevant code at github.com/Darkbblue/dit-lsc, providing a foundation for validating this mechanism across different tasks and scales. For practitioners focused on diffusion model interpretability and multi-task unification, this is a direction worth following closely.