Full Switch to Frontier LLMs for Robot Control: Real-World Customer Deployment Lessons

From VLA to Frontier LLMs: A Bold Architectural Switch

A developer shared on Reddit that over the past month, they completely switched to using frontier LLMs to control robots in customer deployments. This decision was not a minor tweak—it was a complete replacement of their original technical approach.

In the field of robot control, Vision-Language-Action models (VLAs) have long been the mainstream choice. But this developer found that agents built on frontier models like Opus and Astra are generations ahead of any VLA in high-level reasoning capability. This allowed them to build a classic System 1/System 2 architecture—but in the developer's words, "on steroids."

VLA (Vision-Language-Action Model) is a class of end-to-end models designed specifically for robot control. It typically takes visual observations and natural language instructions as input and directly outputs robot actions. Representative examples include RT-2 and Octo. The advantage of VLAs lies in unifying perception, language understanding, and action generation within a single network, reducing information loss between modules and performing well on specific tasks. However, VLAs typically require large amounts of robot demonstration data for fine-tuning. Generalizing to new tasks or environments often requires retraining or additional data, and their reasoning capability is constrained by model scale and training distribution. In contrast, frontier LLMs (such as Opus and Astra), while not specifically trained for robot actions, excel at general reasoning, task decomposition, and common-sense understanding thanks to massive internet-scale data and enormous parameter counts. They can therefore serve as the robot's "brain," handling high-level planning while delegating low-level control to other modules.

Architecture Breakdown: The Synergy of Slow Thinking and Fast Execution

High-Level Reasoning Layer: Frontier LLM Agents

The system's high-level reasoning layer is powered by agents based on Opus and Astra. These two models possess strong general reasoning capabilities, capable of handling complex environmental understanding and task planning. Unlike VLAs trained specifically for robots, these frontier models have been honed through massive data and compute, giving them a natural advantage in generalization.

Fast Execution Layer: Model-Generated Computer Vision Code

The fast execution layer adopts a clever approach: computer vision code written by the model itself handles real-time control. This code is fast, editable, and highly reliable. Compared to end-to-end neural network control approaches, this "code-as-policy" method allows developers to directly review and modify execution logic, making debugging far more intuitive.

The System 1/System 2 architecture originates from psychologist Daniel Kahneman's dual-system theory: System 1 handles fast, intuitive, automatic reactions, while System 2 handles slow, deliberate, logical reasoning decisions. In robotics, this architecture is typically mapped as: a high-level layer using large models for task planning and environmental understanding (System 2), and a low-level layer using fast controllers to execute specific actions (System 1). The innovation here is that System 2 uses frontier LLM agents, while System 1 employs model-generated computer vision code rather than traditional neural network policies. This "code-as-policy" approach gives the execution layer interpretability and editability while maintaining real-time performance.

Unexpected Gains: In-Context Learning and Continual Learning

After switching to frontier LLMs, the team also gained two additional benefits.

In-context learning is a fundamental capability of LLMs, but it is especially valuable in robot control scenarios—robots can quickly understand new tasks from just a few examples without retraining.

Continual learning is equally critical: whenever a new situation arises, the large model steps in and adjusts the entire pipeline. This means the system can continuously evolve over deployment time, rather than requiring manual data re-labeling and retraining as traditional approaches do.

In-context learning refers to a large model's ability to understand and execute new tasks through just a few examples in the prompt without updating its weights. In robot control, this means operators can quickly teach robots new skills through natural language or demonstrations without collecting large amounts of data for retraining. Continual learning refers to a system continuously learning and improving from new situations during deployment. Traditional robot systems often require manual labeling of new data and model retraining, whereas frontier LLM agents can dynamically adjust the entire control pipeline when encountering new scenarios—for example, modifying generated vision code or updating planning strategies—achieving self-evolution. Together, these two capabilities reduce maintenance costs and accelerate how quickly robots adapt to new environments.

On the Right Side of the Bitter Lesson

The developer concluded by noting that this choice puts them "on the right side of the Bitter Lesson." The Bitter Lesson, proposed by Richard Sutton, states that AI systems leveraging large-scale computation and general methods will ultimately outperform systems that rely on hand-designed human knowledge.

By adopting frontier LLMs, this team directly benefits from the capability improvements driven by tens of billions of dollars in global investment. With each new generation of models released, their robot system automatically gains upgraded capabilities—this "rising tide lifts all boats" effect is something specialized VLA models struggle to match.

For practitioners in the field of robot control, this case offers a direction worth considering: when general models become powerful enough, are models trained specifically for particular tasks still necessary?

The Bitter Lesson was proposed by AI pioneer Richard Sutton in 2019. Its core insight is that in the long run, AI systems that leverage large-scale computation and general learning methods (such as search and learning) will ultimately outperform systems that rely on hand-designed features based on human domain knowledge. This lesson has been repeatedly validated throughout AI's history—for example, in speech recognition, computer vision, and Go, where general methods eventually surpassed expert systems. The team in this article chose frontier LLMs as the core of their robot, betting on the scaling effects of general models: as global investment in LLMs continues at enormous scale, model capabilities keep improving, and robot systems automatically gain capability upgrades without specialized training. This requires developers to abandon the obsession with hand-optimizing for specific tasks and instead build architectures that can fully leverage the progress of general models.

One-line Summary

The team completely replaced VLAs with frontier LLMs to control robots in customer deployments, building a System 1/2 architecture with continual learning capabilities.

Paragraph Summary

A developer shared that they completely switched to using frontier LLMs (such as Opus and Astra) to control robots in customer deployments, replacing traditional Vision-Language-Action models (VLAs). They built a System 1/System 2 architecture: high-level LLM agents handle reasoning and planning, while model-generated computer vision code executes real-time control at the low level. This approach brings in-context learning and continual learning capabilities, enabling the system to quickly adapt to new tasks and automatically evolve over deployment time. The team believes this aligns with the Bitter Lesson, allowing them to continuously benefit from massive global investment in frontier models, and raises questions about the necessity of specialized VLA models.