EBench
A benchmark for evaluating the ability of multimodal large models to act as agents controlling robots, comparing general-purpose MLLMs with post-trained VLA/WAM policies under direct-control and harness-invocation modes.
Timeline (last 90 days)
The EBench evaluation report centers on GPT-6-Astra, building an agent that controls a robot through an agent harness and comparing it using the exact same visual and proprioceptive observation protocol as the post-trained policies.
In direct control mode, GPT-6-Astra's overall performance is competitive with post-trained policies, leads on short-horizon tasks, and shows the smallest performance fluctuation across generalization dimensions.
In direct control mode, GPT-6-Astra lags behind post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-intensive tasks.
When the agent is allowed to invoke post-trained VLA or WAM policies, overall task success rate rises to the level of the strongest comparison policy, while token cost drops significantly.
All Facts (4)
The EBench evaluation report centers on GPT-6-Astra, building an agent that controls a robot through an agent harness and comparing it using the exact same visual and proprioceptive observation protocol as the post-trained policies.
50%UnverifiedIn direct control mode, GPT-6-Astra's overall performance is competitive with post-trained policies, leads on short-horizon tasks, and shows the smallest performance fluctuation across generalization dimensions.
50%UnverifiedIn direct control mode, GPT-6-Astra lags behind post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-intensive tasks.
50%UnverifiedWhen the agent is allowed to invoke post-trained VLA or WAM policies, overall task success rate rises to the level of the strongest comparison policy, while token cost drops significantly.
50%