BEAKER
A benchmark for evaluating the embodied action capabilities, knowledge, experimental operations and reasoning abilities of multimodal large models in self-driving chemistry laboratories. It contains 1,000 expert-annotated question-answer samples covering 11 categories of image and video tasks.
Timeline (last 90 days)
The researchers proposed the BEAKER benchmark to evaluate the embodied action ability, knowledge, experimental operations and reasoning of multimodal large models in self-driving chemistry laboratories.
BEAKER contains 1,000 question-answer samples manually created and reviewed by experts in chemistry and embodied AI, covering 11 image and video task types, with no AI-assisted generation or annotation used.
BEAKER's task types include affordance localization, trajectory planning, equipment/state/safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning.
The researchers evaluated 28 multimodal models in an offline, single-turn question-answering setting, covering general-purpose, science-oriented, chemistry-specific and embodied MLLMs.
The evaluation results show that the evaluated models can follow the rough experimental steps in a video and understand the macroscopic order, that is, coarse-grained procedure understanding is acceptable.
The evaluation results show that models struggle to ground actions onto operable regions and feasible trajectories, with fine-grained grounding ability severely lacking.
The evaluation results show that models are not precise enough in fine-grained recognition of laboratory vessels, states and safety regulations.
The evaluation results show that in procedures with slightly more steps, models easily scramble the order, struggling to sequence medium-length procedures.
BEAKER's offline, single-turn question-answering evaluation setting facilitates large-scale, reproducible comparison of model capabilities, but it will underestimate or overestimate real deployment performance; its scores should be understood as an upper-bound estimate of models' embodied cognition rather than a direct prediction of actual laboratory operation success rates.
BEAKER's design approach suggests that evaluation of embodied intelligence needs to be deeply tied to specific experimental scenarios, physical constraints and safety regulations, rather than relying only on generic visual question answering.
All Facts (10)
BEAKER's design approach suggests that evaluation of embodied intelligence needs to be deeply tied to specific experimental scenarios, physical constraints and safety regulations, rather than relying only on generic visual question answering.
50%UnverifiedBEAKER contains 1,000 question-answer samples manually created and reviewed by experts in chemistry and embodied AI, covering 11 image and video task types, with no AI-assisted generation or annotation used.
50%UnverifiedBEAKER's task types include affordance localization, trajectory planning, equipment/state/safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning.
50%UnverifiedThe researchers evaluated 28 multimodal models in an offline, single-turn question-answering setting, covering general-purpose, science-oriented, chemistry-specific and embodied MLLMs.
50%UnverifiedThe evaluation results show that the evaluated models can follow the rough experimental steps in a video and understand the macroscopic order, that is, coarse-grained procedure understanding is acceptable.
50%UnverifiedThe evaluation results show that models struggle to ground actions onto operable regions and feasible trajectories, with fine-grained grounding ability severely lacking.
50%UnverifiedThe evaluation results show that models are not precise enough in fine-grained recognition of laboratory vessels, states and safety regulations.
50%UnverifiedThe evaluation results show that in procedures with slightly more steps, models easily scramble the order, struggling to sequence medium-length procedures.
50%UnverifiedBEAKER's offline, single-turn question-answering evaluation setting facilitates large-scale, reproducible comparison of model capabilities, but it will underestimate or overestimate real deployment performance; its scores should be understood as an upper-bound estimate of models' embodied cognition rather than a direct prediction of actual laboratory operation success rates.
50%UnverifiedThe researchers proposed the BEAKER benchmark to evaluate the embodied action ability, knowledge, experimental operations and reasoning of multimodal large models in self-driving chemistry laboratories.
50%