BEAKER: A New Benchmark for Evaluating Embodied AI Capabilities in Self-Driving Chemical Laboratories

Self-Driving Chemical Laboratories (SDCLs) are driving chemistry experiments toward embodied automation: machines must perceive experimental scenes, infer experimental states, and execute operations under physical, procedural, and safety constraints. Multimodal large language models (MLLMs) are increasingly regarded as the "embodied brain" of such systems, yet existing laboratory benchmarks mostly focus on safety, anomaly detection, or general scientific understanding, leaving their embodied capabilities for execution-oriented operations significantly underevaluated.

To fill this gap, researchers propose BEAKER—a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in self-driving chemical laboratories. It attempts to answer a key question: how far are the current strongest multimodal models from truly being able to "do experiments by hand"?

What BEAKER Measures: 11 Task Types, 1,000 Expert Samples

BEAKER contains 1,000 carefully curated question-answer samples created by experts, covering 11 types of image and video tasks. These tasks correspond to the key judgments required in the execution phase of chemistry laboratories, including: affordance grounding, trajectory planning, equipment/state/safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning.

All questions and annotations were manually created and reviewed by experts in chemistry and embodied AI, with no AI-assisted generation or annotation. This is especially critical in an era that widely relies on model-generated data—it means the benchmark's "value" comes from the experimental common sense of human experts, not the biases of another model.

Affordance grounding originates from the concept of "affordance" in ecological psychology, referring to the possible actions that objects in the environment offer to an actor. In the context of embodied AI, affordance grounding requires the model not only to recognize "this is a beaker" but also to indicate "the rim area of the beaker is suitable for pouring liquid" and "this gripping position allows a stable grasp." It typically outputs image regions (such as bounding boxes or segmentation masks) and serves as the bridge connecting visual perception and physical manipulation. For self-driving chemical laboratories, the accuracy of affordance grounding directly determines whether a robotic arm can interact with vessels safely and effectively, rather than merely "seeing" them.

Trajectory planning in embodied manipulation refers to generating a motion path for a robotic arm or mobile platform from a start point to a goal that satisfies physical and safety constraints. Unlike path planning in classical robotics, trajectory planning for chemistry experiments must also account for scene-specific constraints such as liquid sloshing, vessel collisions, and heat source avoidance. BEAKER includes this in its evaluation to test whether models can translate high-level instructions like "pipette" into executable spatial action sequences, rather than stopping at language-level step descriptions.

Temporal procedure reasoning requires the model to understand the ordering, parallelism, and dependencies of experimental steps along a timeline. Chemistry experiments often involve stages with strict time windows such as waiting, reflux, and quenching, where incorrect ordering can lead to experimental failure or even safety accidents. This task typically takes video or step sequences as input, examining the model's sensitivity to irreversible orderings like "add acid before water" and its overall ability to sequence medium-length procedures.

Evaluation Results: Can Follow Procedures, but Can't Perform Fine-Grained Actions

The researchers evaluated 28 multimodal models in an offline, single-turn question-answering setting, covering general-purpose, science-oriented, chemistry-specific, and embodied MLLMs. The results reveal a clear capability divide:

• Coarse-grained procedure understanding is passable: Models can follow the general experimental steps in videos and understand the macro-level order of "what to do first, what to do next."
• Fine-grained grounding capability is severely lacking: Models struggle to ground actions onto operable regions and feasible trajectories—they know they "need to pipette" but cannot pinpoint "where to pipette and how."
• Fine-grained recognition is weak: Recognition of laboratory vessels, states, and safety specifications is not precise enough.
• Medium-length procedure sequencing is difficult: In programs with slightly more steps, models tend to scramble the order.

This indicates that current models have largely mastered the "narrative layer" of experiments but lack the "operational layer" perception required for execution. BEAKER thus reveals a clear gap: there is a chasm between coarse-grained procedure understanding and precise, action-based perception.

The MLLMs (Multimodal Large Language Models) mentioned in this article refer to large-scale models capable of processing image, video, and text inputs simultaneously, such as GPT-4V, Gemini, Qwen-VL, etc. They typically use a vision encoder to map images into visual tokens, which are then fed together with text tokens into a language model for joint reasoning. The four categories of models evaluated by BEAKER—general-purpose, science-oriented, chemistry-specific, and embodied—differ mainly in pre-training data and post-training tasks: science-oriented models are additionally injected with paper and chart data, chemistry-specific models are further fine-tuned on molecules, reactions, and experimental text, and embodied models incorporate robot manipulation and first-person view data. Understanding this classification helps interpret the evaluation results—even after domain adaptation, models remain generally weak on action grounding tasks, indicating that the bottleneck lies not in domain knowledge but in spatial and physical perception alignment.

The offline, single-turn question-answering evaluation setting means that the model receives only one fixed question and corresponding image or video at a time, outputs an answer independently, and does not interact with the environment, execute actions, or perform multi-turn error correction. This setting facilitates large-scale, reproducible comparison of model capabilities, but it may also underestimate or overestimate real deployment performance: on one hand, it removes the correction opportunities brought by closed-loop feedback; on the other hand, it avoids the cumulative errors in real robot execution. Therefore, BEAKER scores should be understood as an upper-bound estimate of models' "embodied cognition," not a direct prediction of actual laboratory operation success rates.

Why This Benchmark Matters

For self-driving laboratories, model errors are no longer text-level "irrelevant answers" but may be an incorrect pipetting, an incorrect heating order, or a misjudgment of safety specifications. BEAKER's value lies in shifting the evaluation focus from "does it understand science" to "can it safely and accurately execute experimental operations," providing a diagnostic tool for the subsequent deployment of MLLMs in SDCLs.

Its design philosophy also suggests a direction: evaluation of embodied intelligence cannot rely solely on general visual question answering but must be deeply coupled with specific experimental scenes, physical constraints, and safety specifications. Future model improvements for laboratories should perhaps prioritize supplementing "action grounding" capabilities such as affordance grounding and trajectory planning, rather than continuing to stack procedure description capabilities.

Conclusion

With 1,000 expert-annotated samples and a systematic evaluation of 28 models, BEAKER delivers a pragmatic "health check report": multimodal large language models can serve as commentators in the laboratory, but there is still a clear distance before they become reliable executors. For practitioners concerned with embodied AI and automated scientific research, this benchmark provides a reusable diagnostic coordinate.

Background Supplement

Self-Driving Chemical Laboratories (SDCLs) are experimental systems that integrate automated hardware, robotic manipulation, online characterization, and AI decision-making into a closed loop. Typical representatives include flow chemistry platforms, robotic arm workstations, and "chemistry cloud laboratories." Their core cycle is "design—execute—characterize—learn": AI proposes candidate experiments, robots execute synthesis and characterization, and results flow back for the next round of decision-making. In this framework, MLLMs are expected to take on the role of the "embodied brain," decomposing natural language goals into executable actions and monitoring execution states. The affordance grounding and trajectory planning tested by BEAKER are precisely the capabilities this role must possess when moving from the "planning layer" down to the "execution layer," and are also key bottlenecks for current SDCLs transitioning from structured scripts to open-ended autonomous experimentation.

One-line Summary

BEAKER evaluates 28 multimodal models with a thousand expert samples, revealing they understand procedures but struggle with fine-grained experimental operations.

Paragraph Summary

BEAKER is an embodied capability benchmark for self-driving chemical laboratories, containing 1,000 question-answer samples manually annotated by experts in chemistry and embodied AI, covering 11 types of image and video tasks including affordance grounding, trajectory planning, equipment and safety recognition, apparatus setup, and operation and step reasoning. Researchers evaluated 28 general-purpose, science-oriented, chemistry-specific, and embodied multimodal models in an offline single-turn setting, finding that models can follow coarse-grained experimental procedures but perform poorly on affordance grounding, trajectory planning, and fine-grained vessel/safety recognition, and are also prone to errors in medium-length procedure sequencing. The benchmark shifts the evaluation focus from "does it understand science" to "can it safely and accurately execute operations," revealing a significant gap between procedure understanding and action perception.