NVIDIA Cosmos Explained: How Physical AI Bridges Language, Video, and Action

NVIDIA Research Director Ming-Yu Liu sat down with MLST for an in-depth breakdown of the Cosmos model family — its architecture, design philosophy, and what makes it tick. Cosmos is a world model built for "Physical AI," capable of taking text, video, audio, and actions as unified inputs and outputs, enabling closed-loop simulation for autonomous driving and robotics. This article distills the key technical ideas and application directions from that conversation.

One Model, Three Roles

What makes Cosmos most striking is its use of a single model architecture to handle three distinct types of tasks. At its most basic, the model performs visual-language understanding: ask it "how many cars are in this video" or "did the robot successfully place the apple in the basket," and it answers based on the video content. This "video + text in, text out" combination is essentially a standard vision-language model (VLM).

The real pivot comes when you flip that logic — instead of describing a scene, the model starts generating one. Liu's team calls this capability "Cosmos Dreams," and it's used to build closed-loop simulators. The car-turning video shown at the start of the interview featured road surfaces that were never actually filmed — they were entirely generated by Cosmos. It reads video, simulates the road, and learns to handle object interactions.

This means Cosmos elevates "action" to a first-class citizen. The model can actually execute an action, and its next observation is causally produced by that prior action. This causal feedback loop is what sets Physical AI apart from ordinary video generation models.

Architecture: The Autoregressive Tower and the Diffusion Tower

Liu walked through Cosmos's training pipeline in detail. It starts with a language model — either trained from scratch or initialized from an open-source model — that has a baseline understanding of linguistic structure. A visual encoder is then attached, encoding video or image sequences and feeding them into the language model. This forms a VLM that runs autoregressively, token by token — it "thinks in language and narrates what it sees."

The critical second step is initializing a generator from the pretrained VLM weights. Here lies an important architectural distinction: the VLM is autoregressive, while the generator uses diffusion — it's bidirectional. Liu calls the former the "reasoning tower" and the latter the "generation tower." In the generation tower, every token attends to all other tokens, so the generated video, actions, or audio maintain internal coherence within a chunk, while also drawing on instructions passed down from the reasoning tower to understand what the user wants to generate.

Generator training proceeds in two stages: pretraining for foundational capabilities, followed by mid-training where action data is introduced.

Autoregressive vs. Diffusion — these are the two dominant paradigms in generative AI today, and understanding the difference helps explain Cosmos's design logic. Autoregressive models generate tokens sequentially; each new token can only see what came before it, making them naturally suited for language reasoning tasks that require a linear causal chain — GPT-series models are the canonical example. Diffusion models start from noise and iteratively denoise to produce a complete output; each position can attend bidirectionally across the entire sequence during generation, which generally gives them an edge in tasks requiring global coherence like images and video. Cosmos combines both: the reasoning tower figures out what to generate, and the generation tower renders it with high quality. This division of labor lets the model leverage the semantic reasoning accumulated in language model pretraining while also exploiting diffusion's strength in visual coherence. Initializing the generation tower with pretrained VLM weights also dramatically reduces the data needed to train the generator from scratch.

Aligning Different Modalities Across Time

Multimodal fusion runs into a practical problem: video comes at different frame rates, audio at different frequencies, and action signals have their own temporal cadence. How do you get them all running on the same timeline?

Liu's answer is a "temporal position embedding" scheme that normalizes all signals to a common scale. When tokens represent different time chunks or spatiotemporal blocks, the model knows which tokens correspond to the same point in time and what the relative distance is between different time points. Liu emphasized this is the key enabler for a single model to handle audio, video, and actions simultaneously.

The Synergy of Three Dynamics Models

In robotics, Liu highlighted several important modeling perspectives: forward dynamics (given a starting state and an action, predict the future), inverse dynamics (given observed visual change, infer the action that caused it), and policy (what should the robot do to accomplish a task).

Cosmos trains all three within the same model and imposes an "information bottleneck" — limited model capacity forces them to share representations. Liu pointed out that all three are fundamentally capturing "correlations between observations and actions" — they just approach it from different angles. The paper shows synergy between them: learning one task helps the others.

There's a data asymmetry here worth noting: human-generated video is abundant, while action data is scarce. But the modalities produce "positive transfer" across each other. Liu sees the key as a "shared vocabulary" — the visual patterns of how a human hand manipulates an object are highly similar to how a robot manipulates an object. Even if the action spaces don't map exactly, similar patterns make it easier to generalize to other embodiments, including ones never seen before.

Information bottleneck is a theoretical framework in machine learning, originally proposed by Tishby et al. The core idea: when model capacity is limited, the network is forced to extract the most essential and generalizable representations from the input data in order to serve multiple tasks simultaneously, rather than memorizing task-specific features for each one. In Cosmos, forward dynamics, inverse dynamics, and policy share the same parameters. The limited capacity forces them to learn a "compressed representation" that can serve all three perspectives at once. This aligns with the intuition behind multi-task learning: shared representation layers typically generalize better than independently trained models, because they must capture structure that is common across tasks. For robots, this means that interaction rules learned while manipulating an apple can transfer to manipulating objects the model has never seen — the underlying mechanism for what Liu calls "cross-embodiment positive transfer."

As a Simulator: Policy Validation First

A significant portion of the interview focused on using Cosmos as a simulator for training policies — which naturally raises the classic sim-to-real gap in reinforcement learning, and the risk that a policy might "exploit" the simulator.

Liu's pragmatic take is that neural simulators are better suited, at this stage, to policy validation rather than direct policy training. Model builders produce a large number of checkpoints during a single training run — how do you know which one is better? The ideal would be deploying every policy on a real robot and measuring success rates, but that's prohibitively expensive. If the neural simulator's success rate rankings match the real world — not exactly, just preserving "policy A beats policy B" ordering — you can rapidly narrow down the checkpoints that need real-world testing and dramatically speed up the development cycle.

On the question of simulators being exploited, Liu acknowledged that any simulator — and indeed any deep learning system — can be gamed, but he believes the field will develop regularized training methods to address this as it matures. "I don't know the exact path, but I believe people will find a way."

The sim-to-real gap is one of the core long-standing challenges in robotics: policies trained in simulation often suffer significant performance degradation when deployed in the physical world. Traditional physics engines (like MuJoCo and Isaac Sim) try to close this gap through precise rigid-body dynamics modeling, but accurate simulation of complex physical phenomena like soft bodies, liquids, and cloth remains elusive, and the visual appearance of simulated environments differs substantially from reality. Neural simulators learn world models from real video data, so they naturally approach real-world visual fidelity — but their internal physics isn't driven by explicit equations, making them easier for policies to exploit: the policy can discover predictive blind spots in the neural network that simply don't exist in the real world. Liu's "ranking consistency" validation approach essentially positions the neural simulator as an inexpensive coarse-filter tool rather than a precise judge — sidestepping the need for high simulation accuracy.

Three Sizes and an Open Ecosystem

Cosmos 3 comes in three sizes: Super, Nano, and Edge. Super is the frontier model targeting maximum fidelity; Nano sits in the middle and is easier to adapt to various GPU configurations; Edge is optimized specifically for edge devices like Jetson Thor, Orin, and DGX Spark.

The significance of the Edge model is that "the model lives where the robot lives" — no round-trip to a data center required when executing actions. In real deployments, network connectivity isn't always reliable, and some critical tasks must be completed instantly. Because it's smaller, Edge is also easier to fine-tune; the team published a recipe for fine-tuning the Edge model within a day to improve visual understanding. Liu envisions the three sizes working in concert: complex tasks "call up" more powerful models, while immediate, simple tasks rely on the edge device.

Liu repeatedly emphasized Cosmos's open nature: all models, code, and even portions of the training data are open-sourced. Models and datasets are hosted on Hugging Face, code lives in a GitHub repository, and the repo includes post-training recipes to help developers reproduce results. The team has already done post-training on Cosmos using the ZOE dataset, achieving SOTA-level results on grasp-and-place policy tasks.

What Is a World Model, and Where Is This Going?

When asked "what is a world model," Liu compared it to the early debates over defining AGI — there's still no consensus. His own framing: a world model is "a collection of useful tools." We build models always in service of some goal — maybe predicting the future, maybe understanding why something happened.

Looking ahead, Liu is relatively optimistic about navigation tasks, believing current world models are already good enough there. The real challenge lies in manipulation tasks — complex interactions involving contact, compliance, and deformation. He specifically noted that as humanoid robots enter homes, safety becomes even more critical than in autonomous driving: you won't have a car in your living room, but you will have a humanoid robot moving around your children and pets. Developers can't build every kitchen and scenario for real-world testing — which makes neural simulators, in Liu's view, "the only path" to achieving the development velocity needed.