Is LLM the Wrong Foundation for Robot Intelligence? The Great Debate in Embodied AI

A Challenge from the Robotics Community

While the broader AI industry celebrates the capabilities of large language models (LLMs), robotics researcher Ranjay Krishna has put forward a contrarian view: LLMs may not be the right foundation for robot intelligence at all.

This is a provocative claim. In recent years, "large model + robotics" has become the de facto paradigm for Embodied AI — using LLMs as the "brain" for task planning, reasoning, and decision-making, then driving a physical body to execute.

Background: The LLM-Embodied AI Convergence Large language models (LLMs) are neural networks based on the Transformer architecture, pretrained on massive text corpora — GPT-4 and PaLM are prominent examples. Embodied AI, by contrast, insists that intelligence must be grounded in a physical body that develops cognition through real interaction with the environment — a concept rooted in the "behavior-based robotics" proposed by cognitive scientist Rodney Brooks and others in the 1980s–90s. RT-2 (Robotics Transformer 2), released by Google DeepMind in 2023, was a landmark model that transferred a vision-language model directly to robot action control, demonstrating the potential of migrating language pretraining knowledge to physical manipulation. VLA (Vision-Language-Action) models are the systematic extension of this approach, attempting to use a unified language representation space to bridge visual perception and motor action output.

From Google's RT-2 to various VLA models, language seems to have become the natural bridge between perception and action. What Krishna questions is whether that bridge is even necessary.

Language: An Unnecessary Middleman Between Perception and Action?

Krishna's core argument cuts to the heart of the matter: language may be a superfluous intermediary between perception and action.

He offers a compelling example — when humans make physical responses, they don't first "translate" each interaction into language. When you catch a flying ball, yank your hand away from a hot surface, or navigate around obstacles while walking through a room, all of this happens through a direct connection between perception and action — no inner monologue, no verbal description required.

This isn't merely intuitive observation; it has deep neuroscientific grounding. In the human motor system, reflex arcs in the spinal cord can complete stress responses in roughly 20 milliseconds without engaging the cerebral cortex, while the cerebellum handles fine motor coordination and predictive control — both entirely bypassing any language processing pathway.

In other words, if the vast majority of human motor intelligence bypasses language, does forcing robots to "think in language first, then act" impose an unnecessary burden on the system? This could not only introduce latency but may fundamentally distort the structure that intelligence ought to have.

From Perception Directly to Action: The Logic of End-to-End Architectures

Based on this reasoning, Krishna argues that robot models should work the same way: mapping directly from visual and sensor inputs to actions, rather than embedding an LLM in the middle.

This implies a more radical end-to-end architecture — raw signals from cameras, force sensors, tactile feedback, and more feed directly into the model, which outputs motor commands, with the entire loop bypassing the layer of linguistic representation.

Deep Dive: The Technical Roots of End-to-End Learning End-to-end learning is one of deep learning's core paradigms — mapping raw inputs directly to final outputs without relying on hand-engineered feature extraction or modular decomposition. In robot control, this approach traces back to NVIDIA's 2016 paper on "End-to-End Learning for Self-Driving Cars," which output steering angles directly from camera images without explicit perception-planning-control layering. More recently, Stanford's ACT (Action Chunking with Transformers) and Diffusion Policy have further validated end-to-end perception-action learning for dexterous manipulation tasks. The core advantage of these models is avoiding information loss and error accumulation between modules — but the trade-offs are low interpretability, sensitivity to training data distribution, and difficulty handling complex tasks requiring long-horizon reasoning.

This line of thinking is not new in early deep-learning-era robot control, but its serious revival in an era dominated by LLMs carries significant reflective value.

Language as an Intermediate Layer: The Cost Behind the Benefits

To be fair, using language as an intermediate layer is not without merit. It brings interpretability (letting us see what the robot is "thinking"), generalization (leveraging common sense acquired by LLMs from vast text), and task composition (flexibly orchestrating complex tasks via natural language instructions). These are the fundamental reasons VLA-type models have garnered so much enthusiasm.

But Krishna's critique reminds us that these advantages come at a cost. Language is a discrete, low-bandwidth, highly abstract representation, while physical-world interaction is typically continuous, high-dimensional, and sub-symbolic.

Technical Note: Sub-Symbolic Representations and Continuous Control Sub-symbolic representation, as opposed to symbolic representation (such as text or logical propositions), refers to encoding information as continuous numerical vectors without relying on discrete semantic units. In robot action control, sub-symbolic information is especially critical: the process of a robotic arm grasping an object involves high-dimensional continuous signals — joint angles, end-effector torque, contact area — sampled at frequencies typically between 500Hz and 1000Hz. Language, by contrast, has extremely low information bandwidth — human speech conveys information at roughly 39 bits per second, while the control signal bandwidth of a six-axis robotic arm can reach thousands of bits per second. Forcibly mapping high-frequency continuous control signals into the low-bandwidth space of language not only creates a theoretical information bottleneck but also introduces the tens to hundreds of milliseconds of latency inherent in LLM inference — a near-fatal flaw for dynamic tasks requiring real-time feedback, such as catching a ball or emergency obstacle avoidance.

Compressing an action like "catching a ball" — which involves millisecond-level timing and continuous force control — into the representational space of language almost certainly loses critical information and fails to meet real-time control requirements.

A Clash of Two Philosophies of Intelligence

From a broader perspective, this debate is a direct collision between two philosophies of intelligence:

• The Symbolic AI Tradition: Holds that the core of intelligence is symbolic manipulation and reasoning; language is the shortcut to general intelligence, and LLMs are its best current vehicle. This tradition traces back to Newell and Simon's "Physical Symbol System Hypothesis," which posits that any sufficiently complex symbol-manipulating system has the necessary and sufficient conditions for producing intelligent behavior. The rise of LLMs is seen by some as a contemporary revival of this lineage.
• The Embodied Cognition Stance: Holds that intelligence is rooted in the direct interaction between body and environment; motor intelligence evolved before linguistic intelligence, and the two should not be conflated.

Philosophical Background: Embodied Cognition Theory Embodied Cognition is an important school in cognitive science, grounded in Maurice Merleau-Ponty's phenomenological philosophy and Francisco Varela's "enactive cognition" theory. Its central claim is that cognition is not isolated symbolic computation occurring in the brain, but the product of continuous dynamic interaction among the body, the nervous system, and the environment. From an evolutionary perspective, motor intelligence preceded language ability by hundreds of millions of years — fish developed complex motor coordination systems 350 million years ago, while human language has existed for fewer than 1 million years. This means treating language as a necessary foundation for motor control represents a fundamental evolutionary inversion of logic.

Krishna clearly stands with the latter. This resonates with recent academic skepticism about whether large models truly understand the physical world — a model that can write poetry may not know that a glass dropped on the floor will shatter.

Three Takeaways for the Embodied AI Roadmap

This debate has concrete implications for the red-hot robotics sector.

First, layered architectures may be more sensible. A pragmatic middle ground is to let LLMs handle high-level, slow-speed task planning ("go to the kitchen and get a glass of water"), while low-level, fast motor control is delegated to specialized policy networks that don't rely on language. Language only appears where abstract reasoning is needed — not woven throughout the entire control loop.

Practical Example: Engineering Layered Control Layered control architectures have a deep history in robotics engineering. The classic three-layer architecture divides systems into a strategic layer (task planning, second-to-minute response), a tactical layer (behavior coordination, hundred-millisecond response), and an execution layer (low-level motor control, millisecond response). Google's SayCan, released in 2022, was an early template for introducing LLMs into high-level planning — the language model only generates feasible action sequences, while pretrained low-level skill policies handle actual execution, achieving notable results in real kitchen environments. This "slow thinking + fast reaction" division of labor has an interesting structural parallel to psychologist Daniel Kahneman's dual-process theory (System 1: fast, intuitive; System 2: slow, deliberate), and closely mirrors the actual engineering approach of leading companies like Boston Dynamics.

Second, data format determines model format. One of the core bottlenecks in robotics is the lack of large-scale, high-quality action data. Insisting on a language-centric approach means every piece of action data must be paired with text annotations — expensive and potentially bias-inducing. A direct perception-action learning paradigm can, in theory, make fuller use of raw interaction data. Initiatives like Open X-Embodiment are attempting to accumulate diverse physical interaction experience without relying on dense language annotation.

Third, don't be seduced by the "brain" metaphor. Calling an LLM the robot's "brain" is a tempting but dangerous analogy. Biological motor control relies heavily on the cerebellum, spinal cord, and other pathways that require no "thinking." Robot systems likewise don't need to stuff all intelligence into a single language model.

Conclusion: The Foundation of Intelligence — Words or Perception?

Krishna's view may not be the final answer, but it is a necessary wake-up call. Under the halo of LLMs, the industry can easily fall into a path dependency of "everything can be languagified." True embodied intelligence may require us to rethink a fundamental question: Should the foundation of intelligence be built from words, or paved by the shortest, most ancient pathway between perception and action?

For practitioners exploring the future of robotics, this question deserves repeated reflection. The answer is likely not "either/or," but finding the right division of labor between language and direct control — letting language do what it does best (abstract planning, commonsense reasoning, human-machine interaction), and letting the perception-action loop do what it does best (real-time response, continuous control, physical adaptation). This fine-grained division of capabilities may well be the core challenge that embodied AI truly needs to solve in its next phase.