How Do Frontier Agents Control Robots? EBench Tests GPT-6-Astra

Evaluation Background: Can MLLMs Directly Control Robots?

Frontier multimodal large language models (MLLMs) have demonstrated the potential to control robots without task-specific robot training, but where exactly do their capabilities end compared to post-trained Vision-Language-Action (VLA) models and World-Action Model (WAM) policies? This is precisely the question the EBench evaluation report attempts to answer.

Centered on GPT-6-Astra, the report builds an agent that controls robots through an agent harness and compares it against post-trained policies using an identical visual and proprioceptive observation protocol. The study examines two configurations: direct control (the agent directly generates waypoint commands) and harness with policies (the agent can invoke post-trained VLA or WAM policies to execute actions).

VLA (Vision-Language-Action) models are a class of multimodal models post-trained on robot demonstration data that take visual observations and language instructions as input and directly output action sequences. Unlike general-purpose MLLMs, VLAs are fine-tuned on large-scale robot trajectories, coupling "understanding the scene" with "generating executable actions" within a single network, which makes them more robust on contact-rich operations requiring high-frequency closed-loop feedback. The trade-off is that generalization is constrained by the training distribution, making them prone to failure when encountering novel objects or scenes.

WAM (World-Action Model) focuses on learning environment dynamics: the model predicts future observation changes caused by actions and then plans actions accordingly. It typically acquires the ability to imagine physical consequences through video prediction or latent dynamics modeling, providing more coherent action grounding in long-horizon tasks, though inference overhead and error accumulation are its main challenges. The key difference between both and general-purpose MLLMs is that they are aligned to the action space, with outputs falling directly within the robot's controllable command space.

Direct Control: Strengths Stand Out, Weaknesses Equally Clear

In direct control mode, GPT-6-Astra's overall performance is competitive with post-trained policies, particularly leading on short-horizon tasks, and showing the smallest performance variance across generalization dimensions—indicating that frontier MLLMs have good stability when dealing with distribution shifts.

But the weaknesses are equally clear: it lags behind post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-intensive tasks. This reveals a key insight—general-purpose large models excel at "understanding and planning," while fine-grained closed-loop control still relies on specially trained robot policies.

The distinction between short-horizon and long-horizon tasks is typically based on the number of decision steps and sub-goals required to complete the objective. Short-horizon tasks (e.g., grasping a specified object, simple placement) require only a few action steps and mainly test perceptual localization and single-step decision-making—areas where general MLLMs' semantic understanding advantages suffice. Long-horizon tasks (e.g., tidying a tabletop, completing multi-step operations in sequence) require continuous sub-goal decomposition, state tracking, and error recovery; any deviation in one step accumulates downstream, sharply increasing demands on closed-loop control precision and stability—precisely the weak point of general models without robot data post-training.

Contact-intensive tasks refer to assembly, insertion/extraction, wiping, and other operations requiring precise force control and high-frequency feedback. The difficulty lies in visual resolution being insufficient to support sub-centimeter alignment, necessitating real-time adjustment based on proprioception, while general MLLMs' inference frequency is typically far below the control frequency such tasks require.

Harness + Policies: Complementary Synergy, But at a Cost

When the agent is allowed to invoke post-trained VLA or WAM policies, the complementary strengths of both components combine: overall task success rate rises to the level of the strongest comparison policy while token cost drops significantly. This suggests that having the large model handle high-level decision-making while dedicated policies handle low-level execution is a promising architectural path.

However, this mode still falls short of the per-task performance ceiling achieved when each component runs independently. The report identifies three causes of performance loss: suboptimal policy invocation, rigid instruction execution lacking sub-goal decomposition, and interference with grasping when the agent resumes control.

Agent harness refers to the runtime framework wrapping the large model, responsible for registering and invoking tools/policies, managing context, and switching control between the planner and executor. In this evaluation, the harness allows the MLLM to delegate a segment of action execution to a VLA or WAM policy and retake decision-making after the policy returns. Token cost refers to the total model input/output tokens consumed per decision step by the agent; in direct control mode the model must generate actions waypoint by waypoint, resulting in long sequences and high cost, whereas invoking a policy requires only a few high-level commands, significantly reducing total token consumption.

Sub-goal decomposition is the core mechanism of long-horizon tasks: breaking "tidy the tabletop" into independently executable segments such as "find the cup—grasp—move to target position—release," keeping each segment within the short-horizon range. Without decomposition, the policy can only execute a fixed full-segment instruction and cannot adjust based on actual state mid-execution—one of the reasons the harness mode falls short of the per-task performance ceiling.

Capability Boundaries and Design Implications

This study characterizes the capability boundaries of frontier MLLM agents: they have advantages in short-horizon tasks and generalization stability, but still need dedicated policies to fill gaps in long-horizon, high-precision operations. For future agent harness design, the report recommends optimizing three directions—policy invocation decisions, sub-goal generation, and control handoff—to narrow the gap with the theoretical upper bound.

For more details, see the project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/