VITAL: Letting Visual Motion Directly Drive Driving Decisions

When Action Prediction No Longer Depends on Handcrafted Trajectories

The planning module of autonomous driving has long relied on an implicit assumption: trajectories can be defined by independent numerical coordinates, decoupled from visual perception. But driving behavior in the real world is fundamentally a direct consequence of visual motion — the lead vehicle decelerating, a pedestrian crossing, curvature changes in a bend. These visual signals are the root cause that triggers actions.

A paper titled "VITAL: From Visual Motion to Precise Action" approaches the problem from exactly this angle, attempting to answer a question: what happens if action targets are derived directly from visual motion, rather than sampled from a handcrafted trajectory space?

The Precision Ceiling of Trajectory Reconstruction

The research team first examined a simplified scenario: if trajectories are reconstructed directly from recorded future images, can sufficient precision be achieved?

The answer: it captures the main driving maneuvers, but it is not precise enough. This visual reconstruction approach can capture dominant maneuvers such as lane changes and turns, but it exhibits systematic errors at the centimeter level. For low-speed scenarios this may be negligible, but at high speeds, a few centimeters of deviation, amplified over time, can mean a collision risk with an adjacent vehicle.

This finding reveals a key contradiction: visual motion contains the core information of driving intent, but visual reconstruction alone cannot meet the control precision requirements.

VITAL's Action Tokenizer Design

The core idea of VITAL (Vision-Induced Tokens for Action Learning) is to make visual motion the primary prediction target, while using a compact residual branch to handle precision correction.

Specifically, the action representation is split into two parts:

• Visual motion scaffold: a coarse action direction, primarily visual-motion-based, derived from the current image and ego state
• Residual encoding: records the trajectory error not covered by the scaffold and normalizes it to an independent learning scale

The ingenuity of this design is that it allocates a separate, normalized learning space for centimeter-level corrections. If all precision requirements were compressed into a single unified representation, the model would easily be caught between coarse-grained actions and fine-grained corrections, unable to do justice to either.

The entire action expert module is trained using flow matching, and at inference time relies only on the current image, ego state, and navigation instruction — no future image generation is required.

Flow matching is a class of generative model training frameworks that, like diffusion models, belongs to continuous-time generative methods. However, its learning objective is not step-by-step denoising, but directly regressing the continuous probability flow velocity field from the noise distribution to the data distribution. Whereas diffusion models typically require dozens of iterative sampling steps, the flow field in flow matching, once sufficiently trained, can often be approximated with very few steps (even a single step) of integration — this is the underlying reason VITAL can achieve a precision score close to ten-step inference with a single action denoising pass. In action generation scenarios, flow matching treats the "future trajectory" as a continuous distribution, and the model learns a vector field from a simple prior to that distribution, naturally supporting few-step sampling and efficient end-to-end inference.

The Speed Advantage of Single-Step Inference

On the navtest benchmark of NAVSIM v1, VITAL achieves 90.8 PDMS with just a single action denoising step, only 0.13 points behind the ten-step inference score, while end-to-end inference speed is improved by 3.29x.

This data point is worth attention because it means the action expert does not need multi-step denoising iterations like image generation models. The single-step sampling capability of flow matching is fully utilized here, and the gains in inference efficiency come with almost no sacrifice in precision.

Without retraining, VITAL achieves 91.1 EPDMS on NAVSIM v2 navtest and 38.6 on navhard. The performance when transferred to new benchmarks shows that the representation approach of the visual motion scaffold has a degree of generalization ability, rather than overfitting to a specific dataset.

PDMS and EPDMS are comprehensive scoring metrics used in the NAVSIM benchmark to measure planning quality. PDMS (Predictive Driver Model Score) is typically a weighted combination of multiple sub-metrics, covering collision avoidance, drivable area compliance, comfort, progress, and other dimensions; higher values indicate safer planning that is closer to human driving behavior. EPDMS is an extended version introduced in NAVSIM v2, adjusting or adding evaluation dimensions to cover more complex scenarios. navhard is a more difficult test subset within NAVSIM, specifically containing interaction-dense, higher-risk scenarios, so the score of the same model on navhard is usually significantly lower than on navtest. Understanding these metrics helps assess the comparability and difficulty differences between "90.8 PDMS", "91.1 EPDMS", and "38.6" in the text.

Reinforcement Fine-Tuning Supported by Residual Coordinates

Another noteworthy design of VITAL is that the residual coordinate system also supports bounded reinforcement post-training.

Since the residual has already been normalized to an independent scale, reinforcement learning can perform fine adjustments only at the execution level while keeping the visual motion scaffold fixed. This "scaffold-locked + residual fine-tuning" strategy avoids destructive updates to the entire action representation during the reinforcement learning process.

From an engineering perspective, this means the mapping between visual motion prediction and target actions is stable, and reinforcement learning is only responsible for local optimization within the existing framework, rather than searching the action space from scratch.

Bounded reinforcement post-training refers to imposing constraints on policy updates during the reinforcement learning fine-tuning stage, so that action outputs do not deviate too far from the existing representation. In VITAL, since the residual has been normalized to an independent learning scale, reinforcement learning only needs to adjust the residual component under the premise of a fixed scaffold, and the update magnitude is naturally limited to a controllable range. This approach is similar to the KL constraint on policy deviation from the reference model in RLHF, with the goal of using reward signals to optimize execution details while preventing damage to the stable visual motion mapping learned during pre-training, avoiding policy collapse or catastrophic forgetting.

The Significance of Visual Motion as the Primary Prediction Target

The core contribution of VITAL is not the improvement of any single metric, but rather its reorganization of the representational structure of action learning.

The traditional approach is to first define a trajectory space and then have the model fit it; VITAL instead first makes visual motion the prediction target, and then uses residuals to fill the precision gap. This reversal of order may look like an engineering detail, but it changes the model's learning priorities — visually salient motion changes are captured first, while precise numerical alignment is handled as an independent subtask.

For autonomous driving planning, this decomposition may align better with the actual structure of the problem than an end-to-end unified representation. After all, when human drivers change lanes, what they first perceive is the visual motion pattern of the adjacent lane, not a set of coordinate values computed in their head.

More details can be found on the project page: https://worldbench.github.io/vital