NVIDIA Cosmos Lab on Open Models and the Future of Physical AI

AI is moving from the cloud into robots, vehicles, and other physical systems. When models need to understand real-world rules like gravity, friction, and collisions, training on text alone is no longer enough. In a recent episode of the Practical AI podcast, hosts Daniel Whitenack and Chris Benson sat down with Ming-Yu Liu, Vice President of NVIDIA Cosmos Lab, to discuss open models, world models, and what physical AI systems still need before they can truly be deployed.

Why Open Models Matter Especially for Physical AI

Throughout the conversation, Ming-Yu Liu repeatedly emphasized one point: the barrier to innovation in physical AI is higher than in cloud AI. Training a model that can control a robotic arm or drive autonomously requires massive amounts of sensor data, simulation environments, and hardware-in-the-loop testing—resources that not every team can afford.

Open models play the role of infrastructure here. They allow smaller research teams and startups to avoid building foundation models from scratch, and instead fine-tune existing weights and architectures for their specific use cases. This model has already been validated in the language model space—the open source community has built a vibrant ecosystem around models like Llama and Mistral—and Liu believes physical AI will follow a similar path, but at a slower pace, because validating in the physical world costs far more than running a benchmark.

NVIDIA's own Cosmos platform is a product of this approach. It provides world foundation models for physical AI, with the goal not of selling directly to end users, but of enabling developers to build their own systems on top of it.

World Models: Giving AI "Common Sense" About the Physical World

Another core concept in the conversation is the World Model. Similar to how language models predict the next token, world models predict what will happen next in the physical world—how a cup will fall when pushed off a table, how far a car will slide when braking hard on a wet road.

Liu explained that the value of world models lies in providing a learnable physical intuition. Traditional robot control relies on precise physics engines and hand-crafted modeling, but the complexity and noise of the real world far exceed what any simulator can cover. By learning from large volumes of video and sensor data, world models can capture these hard-to-explicitly-model regularities in a data-driven way.

This is especially critical for sim-to-real transfer. If a world model is accurate enough, developers can train policies in virtual environments and then deploy them to real hardware, dramatically reducing trial-and-error costs.

What is sim-to-real transfer

Sim-to-real transfer is a core challenge in robot learning: policies trained in simulators often fail when deployed to real hardware due to factors like physics parameter deviations, sensor noise, and visual texture differences. Common countermeasures include domain randomization (randomizing parameters like gravity, friction coefficients, and lighting during training to make policies robust to variation) and domain adaptation (using a small amount of real data to align simulation and real distributions). World models offer a new path: instead of relying on manually tuned physics engines, they learn dynamics directly from real video, generating training signals that more closely match real distributions and thereby narrowing the gap between simulation and reality.

The Capability Ladder and Bottlenecks of Physical AI

When the topic turned to "what it takes to build increasingly capable physical AI systems," Liu mentioned several levels of capability progression: from single-step perception (recognizing objects, estimating poses), to short-horizon planning (grasping, obstacle avoidance), to long-horizon task execution (multi-step assembly, navigation). The difficulty at each level doesn't grow linearly—it requires models to scale simultaneously across time scales and levels of abstraction.

The current bottleneck is concentrated in data rather than compute. The internet has vast amounts of text and images, but physically annotated interaction data is extremely scarce. Liu believes synthetic data and data generated by world models will play an increasingly important role in the future, but how to ensure the physical realism and diversity of synthetic data remains an open question.

Another challenge mentioned is inference efficiency. Physical systems are extremely sensitive to latency—a self-driving car can't wait for the cloud to return results before making decisions. This means physical AI inference must run efficiently on edge devices, imposing requirements on model compression and hardware co-design that differ from cloud AI.

Why physical AI inference must run at the edge

Physical systems tolerate latency in milliseconds. Force control loops for robotic arms typically need update rates at the 1kHz level, and the perception-decision-control loop for autonomous driving must complete within tens of milliseconds. If inference is offloaded to the cloud, network round-trip alone consumes tens to hundreds of milliseconds, and connection reliability cannot be guaranteed. Therefore, physical AI models must be capable of edge deployment, which directly drives demand for model compression techniques like quantization, pruning, and knowledge distillation, and also pushes co-design of GPUs, NPUs, and other hardware for compute density and power efficiency. The "large model + API call" paradigm common in cloud AI essentially doesn't hold in physical scenarios.

Can the Open Ecosystem Replicate the Success of Language Models

Liu is cautiously optimistic about the prospects of open models in physical AI. On the optimistic side, NVIDIA is advancing foundation model openness through projects like Cosmos, and the research community is sharing datasets and simulation environments. On the cautious side, physical AI is far more fragmented than language—different robot configurations, different sensor setups, and different task scenarios can all make models hard to generalize.

This means the open ecosystem for physical AI may not produce a few "unified" foundation models like language models did, but rather several open models coexisting for specific domains (such as indoor robotics, autonomous driving, and industrial manipulation). For developers, choosing which open model to start from is itself a strategic decision requiring judgment.

The full conversation is available on the Practical AI podcast.

The difference between world foundation models and ordinary foundation models

A World Foundation Model is a class of large generative models specifically designed for modeling the dynamics of the physical world, with typical examples including NVIDIA Cosmos and Google's Genie series. It differs from language foundation models in that its inputs and outputs are multimodal physical signals like video, depth maps, and point clouds rather than pure text tokens; its training objective is not predicting the next word but predicting how a scene evolves over time; and its evaluation criteria are not perplexity or benchmark scores but whether the generated content is physically plausible and can effectively support downstream policy learning. These models typically require thousands of hours of driving and robot manipulation video for pre-training, can have billions of parameters, and cost far more to train than most research teams can afford—which is precisely why "open weights" are so critical to their ecosystem.

One-line Summary

NVIDIA's Ming-Yu Liu discusses how open models and world models are advancing physical AI and its data bottlenecks.

Paragraph Summary

In the Practical AI podcast, Ming-Yu Liu, VP of NVIDIA Cosmos Lab, discussed the development path of physical AI. He noted that the barrier to innovation in physical AI is far higher than in cloud AI—training robot or autonomous driving models requires massive sensor data, simulation environments, and hardware-in-the-loop testing, so open models serve as infrastructure, allowing smaller teams to fine-tune existing weights. World models are another core concept, capturing hard-to-explicitly-model physical regularities by learning from video and sensor data, making them especially critical for sim-to-real transfer. The biggest current bottleneck is data: physically annotated interaction data is extremely scarce, and while synthetic data is highly anticipated, its physical realism and diversity remain open questions. Additionally, physical systems demand efficient edge inference, imposing new requirements on model compression and hardware co-design. Liu is cautiously optimistic about the open ecosystem, believing physical AI is more likely to form a landscape of several domain-specific models coexisting.