Fudan's Jia Xiaosong: VLA and WAM Architectures Are Hot, but the Real Bottleneck Is How We Train

The fields of autonomous driving and embodied intelligence are being flooded with new architectures like VLA (Vision-Language-Action models) and WAM (World-Action Models). But Jia Xiaosong, assistant professor at Fudan University's Institute of Trustworthy Embodied Intelligence, offered a counterintuitive take in a recent conversation with Leiphone's Zuo Lin You Li channel: model structure is no longer the most fundamental bottleneck—the real crux lies in how we train, especially the inherent limitations of imitation learning.

From Trajectory Prediction to End-to-End: Why Modularization Has Run Its Course

Jia Xiaosong's research trajectory has tracked almost every technical turn in autonomous driving over the past few years. In late 2021, while pursuing his direct PhD at Shanghai Jiao Tong University, he received a call from his advisor Yan Junchi. He had planned to continue working on trajectory prediction for autonomous driving. Yan's advice was to take a different path: prediction sits between perception and decision-making, constrained by both upstream and downstream, so why not do end-to-end instead.

This judgment was quickly confirmed by Jia's own experiments. He summarized two fatal problems with traditional modular systems.

Information is continuously lost between modules. A pedestrian waving, a leading car's brake lights coming on—these cues, crucial for decision-making, often aren't retained at the upstream perception stage. By the time the prediction module receives them, researchers only have coordinate sequences of vehicles and pedestrians, with no way to use them.

Modules have inconsistent optimization objectives. The perception team spends months raising the detection rate for vehicles 200 meters away, but the decision module for urban driving may not care about vehicles that far at all. Single-module metrics improve, but overall driving performance may not.

The value of end-to-end is letting perception, prediction, and decision-making jointly optimize around the same final objective. In 2022, Jia and collaborators at OpenDriveLab developed the TCP model, fusing the trajectory prediction and control branches with surprisingly good performance; the work was published at NeurIPS 2022. He later reflected that the value of intermediate tasks lies more in providing supervisory signals than in achieving true interpretability—detecting a vehicle doesn't mean downstream decision-making actually cares about it; failing to detect it doesn't mean the end-to-end model internally doesn't know it's there.

End-to-end autonomous driving refers to using a single unified neural network to map directly from sensor inputs (cameras, LiDAR, etc.) to vehicle control commands (steering, throttle, braking), without explicitly splitting perception, prediction, planning, and other independent modules in between. The motivation is to avoid information bottlenecks and objective mismatches in modular pipelines, letting all subtasks jointly optimize around final driving performance. The cost is reduced interpretability, large training data requirements, and difficult debugging; the industry has also seen cases where end-to-end models struggle to pinpoint failure causes. Academia often compares it with modular approaches, which are engineering-controllable and can be verified segment by segment, but whose ceiling is limited by the local optima of each module.

The Ceiling of Imitation Learning: Even Infinite Data Won't Drive Perfectly

End-to-end solved module fragmentation but hit another wall. Around 2022, Jia conducted extensive closed-loop experiments with ThinkTwice in CARLA simulation, collecting what he considered near-infinite data, investing 100TB of storage and substantial compute to train larger models. The results were sobering: as data and model scale grew, driving capability didn't leap from "drives okay" to "drives perfectly."

He attributed the problem to the objective function of imitation learning itself. Imitation learning optimizes how closely model behavior approximates human demonstrations, not how good the decisions themselves are. A typical phenomenon in simulation: the model struggles to perfectly learn "stop at red, go at green" because it can find a simpler correlation—if the car ahead moves, I move. This shortcut is enough to push the loss very low on the vast majority of samples, and the model loses the incentive to optimize the few critical scenarios.

Two derivative problems are equally thorny. First, multi-solution conflict: different drivers take different reasonable actions in the same scenario, and MSE-type losses make the model fit an average action, which may not be reasonable. Second, teaching only right, not wrong: human driving data almost never contains samples of "getting stuck at an intersection and then escaping," so once the model makes a mistake, it doesn't know how to correct it.

Imitation learning is a class of methods that learn policies from expert demonstrations, typically by minimizing the mean squared error between model actions and human actions or maximizing the likelihood of demonstrated actions. Its theoretical weakness is called "covariate shift": during training the model only sees the expert state distribution, and once it makes a mistake and enters states not present in demonstrations, it lacks correction ability and errors accumulate. Behavior cloning is the simplest form; inverse reinforcement learning, DAgger, and others attempt to mitigate the problem through interaction or reward inference. This contrasts with reinforcement learning, which optimizes cumulative reward and allows trial-and-error from failures.

Imitate First, Then Reinforce: From 0 to 90, Then the Last 10

The limitations of imitation learning point to reinforcement learning. Jia's analogy is vivid: imitation learning takes a model from 0 to 90, and reinforcement learning tackles the remaining 10. But the two aren't substitutes—he ran reinforcement learning experiments from scratch and found convergence extremely slow; it could handle dozens of CARLA scenarios but he had no confidence facing thousands of real-world scenarios.

In 2022, he developed DriveAdapter, connecting a perception module to an existing reinforcement learning planner—essentially early two-stage end-to-end training; the work received an ICCV 2023 Oral. Today "imitation pretraining, then reinforcement post-training" has become a common paradigm in large model research, but at the time it was still novel.

The truly difficult part remains how to scale reinforcement learning—getting dozens, hundreds, or even thousands of capabilities of a general model to improve simultaneously rather than at each other's expense. Jia states plainly that this problem remains unsolved today, and the hot on-policy distillation (OPD) in the large model field is pondering something similar.

On-Policy Distillation (OPD) refers to having a student model sample trajectories from its own policy distribution, then having a teacher model provide guidance signals on those states for distillation. Unlike traditional offline distillation, which only uses teacher trajectories, OPD exposes the student to states arising from its own mistakes, thereby mitigating distribution shift. In large model reinforcement learning post-training, it's often used to transfer a teacher model's reasoning ability to a student while reducing dependence on expensive reward models. Challenges include high teacher query costs, student exploration potentially drifting out of valid regions, and how to stably balance distillation loss with task reward.

World Models: Latent-Space Simulation and Behavior/Rendering Separation

Reinforcement learning requires massive trial and error, which real vehicles obviously can't do. Inspired by DeepMind's DreamerV3 mining diamonds in Minecraft, Jia and collaborators developed Think2Drive: training a lightweight world model with neural networks, letting driving policies explore in latent space. Under the experimental setup at the time, one GPU could run 1,000 to 2,000 latent-space environments simultaneously; a single NVIDIA A6000 trained in about three days a policy capable of completing complex tasks in CARLA Leaderboard 2.0.

How to ensure the correctness of latent-space models? His answer is alternating iteration: first train the world model, let the policy explore within it, then test in the external environment; when prediction deviations are found, add new data to update the world model. "Just because a model can predict the future doesn't mean it has grasped the completely correct laws of the world."

The subsequent Bench2Drive-R further split the world model into two layers. The behavior world model handles how other vehicles move, how traffic participants interact, and even actively creates difficult scenarios; the rendering world model generates realistic and consistent images. The reason for separation is practical: when two cars are extremely close, it's hard to judge a collision from video alone, but the underlying position and motion states maintained internally can give a definitive answer.

He categorizes world model uses into three types: as supervisory signals for learning physical representations (common in current WAM research), as simulators supporting reinforcement learning and difficult-scenario evaluation, and as aids for reasoning about the consequences of different action plans at inference time.

But the bottlenecks are equally obvious: long-duration generation produces hallucinations, with the same vehicle jittering between frames; generative world models are slow at inference and compute-expensive, making million-scale parallel environments hard to support. On technical routes, graphics methods like 3D Gaussian Splatting are controllable but anti-Scaling Law, while video diffusion models have generalization potential but single-scenario quality may not surpass finely tuned reconstruction; he judges the future is a combination of both.

DreamerV3 is a reinforcement learning algorithm based on world models proposed by DeepMind. Its core is learning an environment dynamics model in latent space and training the policy entirely on imagined latent-state trajectories, greatly improving sample efficiency. It learned to mine diamonds in Minecraft from scratch, demonstrating the generality of a single hyperparameter configuration across multiple task types. Its structure typically includes representation, transition, reward models, and policy/value networks, alternating world model learning with imagined rollouts during training. Limitations include accumulating long-horizon prediction errors and the need for external interaction to correct deviations between latent-space models and the real environment.

Architecture Isn't the Answer: Data Mixing and Scaling Reinforcement Learning Are

Back to VLA and WAM. Jia observed that these models introducing stronger vision-language capabilities don't necessarily outperform fully optimized end-to-end models on some autonomous driving benchmarks. Passing a sheep, a VLA has deeper understanding of the sheep, but driving performance may not improve. "What truly limits the model is whether it can make correct decisions, not whether it recognizes this thing."

If training remains primarily imitation learning, changing architecture won't bypass imitation learning's inherent problems. He therefore remains restrained about improvements at the network structure level, believing "carving patterns on the circuit diagram" doesn't solve real problems. Valuable directions are: how to select truly valuable data, how to go beyond brute-force imitation learning, how to design more effective reward signals, and how to let a model reinforce thousands of skills simultaneously.

This judgment extends to embodied intelligence. The robotics field is still collecting large amounts of demonstration data and scaling imitation learning; Jia believes this is necessary but will ultimately hit the same bottleneck as autonomous driving. The difference is that autonomous driving has accumulated massive real data, and the problem becomes data mixing—don't let simple repetitive scenarios dominate training; while embodied intelligence hasn't even obtained its "first pot of gold," dexterous hand teleoperation is far harder than grippers, and he chose Ego data precisely to make data collection blend into daily life like autonomous driving.

He also cautions that embodied intelligence is far harder than autonomous driving: contact, friction, and deformable objects are difficult to model precisely, and world models' hallucination and speed problems persist. Autonomous driving methodology can be borrowed, but one shouldn't assume it has solved embodied intelligence's problems.

From a car to a pair of hands, Jia Xiaosong's research journey offers a pragmatic perspective on technological evolution: every turn isn't because a new architecture is trendier, but because old methods hit a wall they couldn't get around.

One-line Summary

Fudan's Jia Xiaosong argues VLA/WAM aren't the bottleneck—training methods and imitation learning limitations are key.

Paragraph Summary

Fudan assistant professor Jia Xiaosong points out in conversation that the real bottleneck in autonomous driving and embodied intelligence isn't model architecture but training methods. He moved from trajectory prediction to end-to-end, noting modular systems suffer information loss and objective mismatch; he then found imitation learning optimizes behavioral similarity rather than decision quality, leading models to learn shortcuts, face multi-solution conflicts, and fail to recover from errors. He advocates imitation pretraining followed by reinforcement post-training, though scaling reinforcement learning remains open. On world models, he proposes separating behavior world models from rendering world models, warning of hallucination and compute bottlenecks. Final judgment: if VLA and WAM still rely on imitation learning, changing architecture won't bypass inherent limitations—data mixing and training methods matter more than network structure.