SMD: Simplified Online Distillation Achieves 4-Step Generation That Surpasses Multi-Step Diffusion Models

Diffusion models and flow matching models have already proven their capabilities in image, video, and 3D content generation, but sampling cost remains an unavoidable pain point when it comes to deployment. Few-step distillation attempts to compress sampling from dozens of steps down to within 4 steps, typically at the cost of introducing auxiliary score models, adversarial training, or special network architectures—these additional components make the training pipeline heavier and also limit the generality of the method.

A recent work proposes Simplified Matching Distillation (SMD), with a rather restrained core claim: simple online velocity regression (on-policy velocity regression) alone is sufficient to support 4-NFE generation quality.

A Distillation Approach That Abandons Auxiliary Modules

Existing few-step distillation methods can be roughly divided into two categories. One type relies on additional score models to estimate the teacher distribution, while the other introduces adversarial losses to bring the student and teacher output distributions closer. Both can improve results, but at the cost of a complex training pipeline, sensitivity to hyperparameters, and often being tied to a specific architecture.

SMD's starting point is to return to the most plain regression objective: have the student model predict the mean of the teacher's velocity field over coarse-grained time steps. This approach itself is not new, but the authors found that, if handled properly, simple online velocity regression is already a strong baseline.

The key issue is that the teacher's true velocity is not constant within each coarse time interval, and directly using the interval mean as supervision would lose information about local variation.

Few-Step Distillation and NFE

NFE (Number of Function Evaluations) refers to the number of times the model needs to call a network forward pass when generating an image (or a video, or a 3D asset). Standard sampling for diffusion models requires 20 to 50 forward passes, because each step only advances a small distance along the learned velocity field, and the step size must be small enough to ensure trajectory stability; the same applies to flow matching models, which, although their trajectories are closer to straight lines, usually still require over a dozen to several dozen integrations. The goal of few-step distillation is to compress NFE from dozens down to 4 steps or even 1 step, reducing inference cost by an order of magnitude. The difficulty here is that the student network must directly predict, on an extremely coarse time grid, the displacement needed to jump from one moment to the next, while the teacher's velocity within this interval is not constant—this is precisely the fundamental reason SMD needs to introduce the Pivot residual proxy.

Pivot: A Residual Proxy During Training

In order to preserve changes in the teacher's velocity during coarse-grained transitions, the authors introduce a residual proxy called Pivot. Its role is to separate local corrections from the student's interval-mean prediction: the student is responsible for predicting the average velocity over the entire interval, while Pivot is responsible for modeling the residual fluctuations within the interval.

This proxy exists only during the training stage and does not change the generator structure at all during deployment. In other words, the computational graph at inference time is no different from that of an ordinary few-step model, with no additional branches and no additional parameters. This is also an important difference between SMD and many distillation methods—it leaves complexity on the training side and simplicity on the inference side.

Velocity Field, Online Regression, and "On-Policy"

The velocity field v(x,t) is the core quantity of flow matching/diffusion models: at state x and time t, it indicates the direction and speed at which the data should move. A common way to train the student is offline regression—first use the teacher to pre-sample a batch of trajectories, then have the student fit these fixed samples. But the student will visit states the teacher has never traversed, leading to error accumulation. Online (on-policy) regression is different: at each step, the student samples a trajectory starting from noise, records the states it reaches along the way, and then has the teacher provide velocities on these states actually visited by the student as supervision signals. In this way, the supervision distribution is aligned with the student's own inference distribution, avoiding the "distribution shift" between training and inference. This is exactly the online approach adopted by SMD, and the "Pivot residual proxy" is an auxiliary prediction head additionally introduced under this framework to absorb velocity fluctuations within the interval. It is invoked only during training, and the student backbone incurs no extra computation at deployment.

A Lightweight Mechanism That Repels Weak Teacher Predictions

In addition to fitting the teacher's velocity, SMD also adds a reverse constraint: keep the student away from the teacher's weaker predictions.

The intuition behind this design is that the teacher model's prediction quality is not uniform across different time steps, and outputs in certain regions are themselves not reliable enough. If the student only blindly fits, it may inherit these undesirable patterns. By applying a repulsion signal, the student is pushed toward the more reliable side of the teacher's predictions, thereby avoiding falling into bad modes.

The authors describe this mechanism as lightweight because it does not require the discriminator commonly used in adversarial training, nor does it require online score estimation; it merely adds an extra constraint to the training objective.

Why "Repulsion" Instead of "Alignment"

Traditional knowledge distillation assumes that the target given by the teacher is always reliable, and the student only needs to approximate it as closely as possible. But in few-step scenarios, the teacher's predictions in certain time intervals (especially at endpoints where t is close to 0 or close to 1) are themselves very noisy, and if the student fits them unconditionally, it will instead solidify this uncertainty. SMD's approach is to add a repulsion term to the loss: when the teacher's prediction confidence is low or its variance is large, keep the student away from this prediction, thereby concentrating the learning signal on the time regions where the teacher is more reliable. This idea shares similarities with negative samples in contrastive learning and with "pushing probability mass away from low-quality regions" in some energy models. Compared with adversarial training, which requires an additional discriminator to estimate real versus fake online, this repulsion based on the teacher's own confidence requires no auxiliary network at all, and the training overhead is almost zero, which is why it is called a lightweight mechanism.

Validation of Cross-Modal Generality

SMD's experiments cover a fairly wide range: image generation and editing, 3D generation, video synthesis, and joint audio-video generation, with model scales extending up to 33B parameters.

Under the 4-NFE setting, SMD surpasses existing strong few-step baselines on multiple benchmarks, and even exceeds the multi-step teacher model on some tasks. This means that the student model, using 4-step sampling, can achieve generation quality that the teacher can only attain with more steps.

The method also supports cross-model distillation, multi-teacher distillation, and causal video distillation, showing that it is not tied to a particular teacher architecture or a particular modality.

Why This Matters

The engineering value of few-step distillation lies in inference cost. When generative models enter the productization stage, the difference between 4 steps and 20 steps is directly reflected in latency and compute bills. The significance of SMD is not just that it is another distillation method, but that it demonstrates a simpler path: no adversarial training, no auxiliary score model, and purely regression-based training can achieve competitive few-step generation results.

For teams that need to deploy diffusion model generation capabilities, the training side can be complex, but the inference side must be simple—SMD's trade-off on this point is more worth referencing than a mere improvement in metrics.

The paper is titled "Simplifying and Improving On-Policy Distillation for Few-Step Diffusion," and its core contributions can be summarized as the joint design of the Pivot residual proxy and the repulsion mechanism, along with the resulting cross-modal and cross-scale scalability.

One-line Summary

SMD uses only online velocity regression plus a Pivot residual proxy to achieve 4-step few-step diffusion generation.

Paragraph Summary

Simplified Matching Distillation (SMD) addresses the high sampling cost of diffusion and flow matching models by proposing an extremely simple few-step distillation scheme: it abandons the auxiliary score models, adversarial training, and special network architectures commonly relied on by existing methods, and relies only on online velocity regression to support 4-NFE generation. Its core design includes two parts: first, the Pivot residual proxy, which models residual fluctuations within coarse time intervals while the student predicts interval means, and exists only during training without changing the generator structure at deployment; second, a lightweight mechanism that repels weak teacher predictions, preventing the student from inheriting outputs from unreliable regions of the teacher. Experiments cover image, 3D, video, and audio-video generation, with scales up to 33B parameters. Under 4-NFE, it surpasses strong few-step baselines and even exceeds the multi-step teacher on some tasks, demonstrating cross-modal and cross-architecture generality.