Inside DeepMind's Robotics Lab: How VLA Models Teach Robots to Think Before They Act

Four Years Later, the Robotics World Has Transformed

In a special episode of the Google DeepMind podcast, host Hannah Fry visited DeepMind's robotics lab in California, guided by Kanishka Rao, Director of the Robotics division. It had been exactly four years since Fry's last visit.

The most striking change was immediately visible: the privacy screens that once filled the lab had been completely removed. "We don't need them anymore — the whole lab is open now," Kanishka explained, attributing this to the stronger visual foundation of today's robot models, which no longer require careful control of lighting or background.

Behind that simple statement lies a long history of hard-won technical progress. Visual generalization has long been one of the most stubborn challenges in robotics — traditional robot vision systems, built on rule-based approaches or shallow neural networks, were extremely sensitive to lighting, background, and object orientation. That's precisely why labs needed privacy screens and tightly controlled shooting environments. Early deep learning approaches improved things somewhat, but models would typically only work reliably in scenes closely resembling their training data — change the room or the lighting, and performance would collapse. It wasn't until large-scale pretrained visual foundation models were integrated into robotic systems that this problem was fundamentally addressed, thanks to the general-purpose feature representations learned from billions of images.

Fry put it bluntly: even though only four years had passed, "in robotics, that's an absolute eternity." The driving force behind this transformation is the rise of large language models and multimodal models.

VLA Models: Turning Physical Actions into Tokens

The centerpiece of the visit was understanding how DeepMind channels the power of large models into physical robots. Kanishka explained that the team wants robots to be "generalizable for human use," which means they must understand universal human concepts. The biggest breakthrough of the past few years has been building robotic systems on top of large Vision-Language Models (VLMs).

These VLMs have "a very good understanding of everything in the world," and the latest robot models are built on this foundation — demonstrating remarkable generalization when encountering new scenes, new visuals, and new instructions.

Making Actions First-Class Citizens Alongside Vision and Language

To achieve this, the team developed VLA (Vision-Language-Action) systems. The core idea: treat the physical actions a robot executes in the real world as equally important as visual tokens and language tokens.

The design of VLA draws inspiration from the unified token-processing paradigm of large language models. In a standard VLM, images are segmented into visual patch tokens and text is encoded as language tokens — both modeled together within the same Transformer framework. VLA's innovation is to also represent physical robot actions (such as joint angles and end-effector displacement) as "action tokens," either discretized or continuous, fed into and output from the same model alongside visual and language tokens. This means the model no longer needs a separate controller to "translate" perception into action — instead, it handles the full chain from perception to action end-to-end within a unified sequence prediction framework. DeepMind's RT-2 series is a landmark example of this paradigm.

With this architecture, the model can analyze action sequences and determine what new sequence of actions to execute when faced with an unfamiliar situation — a capability the team calls "action generalization." This goes beyond simply perceiving the surrounding scene; the model actively plans the actions it's about to take.

From Short-Horizon to Long-Horizon Tasks

Early robot versions could only handle quick, simple tasks — picking up an object and placing it elsewhere, or unzipping a bag. But Kanishka noted that to be truly useful to humans, robots need to handle "long-horizon tasks."

Long-horizon tasks require a system to correctly sequence and execute dozens or even hundreds of sub-actions, where any single failure can cascade into total collapse. The industry and academia broadly address this with hierarchical architectures: a high-level planner decomposes a natural language goal into a sequence of sub-goals, while a low-level controller translates each sub-goal into precise physical actions. This "brain + cerebellum" decoupled design is a close analogy to how the human nervous system works — the prefrontal cortex handles planning while the cerebellum coordinates motor execution.

Today, DeepMind has built an agent that can coordinate fine-grained individual actions and chain them together into complete long-horizon tasks. For instance, if you need to pack for a trip to London, the agent can first look up the London weather forecast, decide what to pack based on that, and even pack your bag for you.

This "build layer by layer" architecture — with a foundation model at the base, additional layers stacked on top, until a series of actions can be chained together to accomplish complex tasks — has become the key to making robots genuinely practical.

Version 1.5: Two Breakthroughs — Agency and "Thinking"

Kanishka revealed that version 1.5 of the robotic system introduced two key capabilities: an agent component and a "thinking" component.

"Thinking" means the robot pre-plans what action it's about to take before executing it — generating thoughts first, then acting. This "think before you act" process makes the robot more generalizable and significantly improves performance.

This principle maps closely to Chain-of-Thought (CoT) reasoning in large language models. CoT was introduced in 2022 by Google researchers, with the core finding that letting a model explicitly output step-by-step reasoning before answering dramatically improves accuracy on complex tasks — believed to work because the intermediate steps provide richer context for subsequent tokens and force the model to allocate more "compute" to hard problems. DeepMind applies this same principle to robotics: the model generates natural language intent descriptions before outputting action tokens, forming a closed-loop "think first, act second" cycle. The success of OpenAI o1 and DeepSeek R1 further validated this direction — a sufficient "thinking budget" is a general-purpose mechanism for raising a model's reasoning ceiling. Kanishka marveled that such emergent properties are "just kind of magical," because what humans do intuitively is, for a robot, extraordinarily difficult.

Live Demos: From Packing Lunch to Sorting Trash

Millimeter-Precision Lunch Packing

The first demo involved packing a lunchbox — which Kanishka called "one of the hardest tasks" because it requires millimeter-level precision: gripping a resealable bag the right way, fitting bread into a tight space, handling a chocolate bar and grapes without crushing them.

Fry recalled her first visit years ago, when the lab was still trying to stack large Lego bricks. "The pile of broken Lego in the corner really told you how hard it was." Today's robots can complete this kind of dexterous manipulation end-to-end, learning entirely from data — primarily collected through teleoperation.

Teleoperation is currently one of the most common data collection methods in robotics: researchers control robots in real time using data gloves, force-feedback handles, or VR devices, while the robot simultaneously records its joint states, end-effector poses, camera images, and force sensor readings — producing paired observation-action trajectory datasets. This approach yields high-quality, natural-looking data that closely reflects genuine human intent. The downside is the high cost: an hour of skilled operation might yield only dozens of usable trajectories. Robots learn from this data in a first-person perspective, gradually internalizing human motion patterns.

Generalization: Handling Objects Never Seen Before

The second demo focused on generalization. Researchers layered Gemini on top of the VLA, allowing users to give natural language instructions — and the robot could even hold a conversation while working. When Fry asked it to place a block "in the style of Batman," the robot politely explained it couldn't roleplay specific characters, but completed the task anyway.

More impressively, Fry produced a stress ball she'd brought with her — an object the robot had never encountered — and the robot still managed to open a green canister's lid, place the pink ball inside, and close the lid again. This kind of "openness" to novel objects is the defining competitive edge of the new generation of general-purpose robots.

Long-Horizon Task: Trash Sorting with Real-Time Rule Lookup

In the trash-sorting demo, Fry gave a high-level instruction: "I'm not sure about the recycling rules in San Francisco — can you look them up and sort this out for me?" The robot first explained that San Francisco requires sorting waste into recyclables, compost, and landfill, then proceeded to sort each item accordingly.

Behind this was an Orchestrator that decomposed the high-level task into smaller instructions, which the VLA then executed end-to-end. The orchestrator plays the role of a high-level planner, leveraging the LLM's powerful commonsense reasoning and task decomposition capabilities, while the VLA acts as the low-level controller focused on dexterous physical execution — a textbook example of a hierarchical agent architecture in a real-world setting.

Dual-System Architecture and "Thought Visualization"

On system architecture, Kanishka described two cooperating modules: an ER model (essentially a VLM) that excels at reasoning and handles coordination, and a VLA model responsible for executing physical actions. Together they tackle long-horizon tasks.

In a humanoid robot clothing-sorting demo, a different design was used — a single model that both thinks and acts within the same loop, with no hierarchical separation. The most striking feature was that viewers could see in real time what the robot was "thinking" — much like watching Gemini display its reasoning process before producing an answer. This "thought visualization" isn't just impressive for demos; it's a critical tool for debugging and improving robot behavior, allowing researchers to inspect intermediate reasoning steps and quickly pinpoint whether a failure occurred in the perception or planning stage.

The Bottleneck Is Still Data — But the Revolution Is Near

Asked whether another revolution was still needed, Kanishka candidly said at least "one major breakthrough" is still required. Even the most advanced robots need enormous amounts of data to learn tasks, and the team urgently needs to find ways for robots to learn more efficiently from less data.

The core bottleneck is the structural scarcity of physical interaction data — internet text data runs to trillions of tokens, image data to billions of images, while high-quality robotic physical interaction data amounts to only a few million trajectories. This gap stems from multiple factors: physical interaction is inherently real-time and embodied — you can't scrape it in bulk like text or images; valid data must include temporally paired sensor-action sequences requiring expensive collection equipment; and the action spaces and sensor configurations vary dramatically across robot platforms, making cross-platform reuse difficult. Researchers still need to collect real-world sensorimotor experience on robots one trajectory at a time.

However, Kanishka pointed to a promising direction: humans have already accumulated vast amounts of manipulation videos and instructional content (think YouTube cooking and repair tutorials), and future robots may be able to learn directly from them. Extracting trainable "implicit action labels" from these human videos — which lack precise joint data — is a current hot research topic.

Beyond data, safety remains a challenge that must be solved. But Kanishka expressed full confidence in the foundational architecture, believing these "core building blocks will guide the ultimate blueprint for general-purpose robotics."

Fry concluded: these robots are sometimes slow, sometimes a little clumsy, but they "can understand semantics, understand the scene in front of them, and reason about complex tasks — something that would have been completely unimaginable just a few years ago." We may well be standing on the eve of a genuine robotics revolution.

Key Takeaways