SciDiagBench: A New Benchmark for Evaluating Scientific Figures by Figure Type

Scientific figures are key carriers for conveying complex information in academic papers, but automatically evaluating their quality has long been a challenge. Existing vision-language model (VLM) evaluation benchmarks often apply generic criteria to all figure types, ignoring the fact that human experts adjust their judging criteria based on figure type during review. The recently proposed SciDiagBench attempts to fill this gap by systematically categorizing figures and incorporating human preference guidance, making automatic evaluation closer to real human judgment.

Limitations of Existing Evaluation Benchmarks

According to the original paper, current VLM-based evaluation benchmarks have two major shortcomings. First, benchmark datasets typically lack systematic categorization of figure types, resulting in all figures being measured against the same set of generic criteria. However, scientific figures themselves span multiple types, each with unique visual structures and content organization—for example, graphical abstracts emphasize visual impact and rapid communication of core conclusions, technical roadmaps focus on logical flow and stage division, while illustrative examples prioritize clear presentation of details. Second, existing evaluation criteria are often heuristically designed without explicit alignment to human evaluation preferences.

These two problems together cause automatic evaluation results to deviate from human expert judgment. In scenarios such as peer review, experts naturally switch their evaluative perspective based on figure type, but machine evaluation cannot do this.

Vision-language models (VLMs) are a class of multimodal models that can simultaneously process image and text inputs and generate text outputs, such as GPT-4V, LLaVA, Qwen-VL, etc. In figure evaluation tasks, VLMs are typically asked to read figure images and output quality scores or comments. However, VLM judging tendencies depend heavily on their training data and prompt design: if the prompt does not specify the figure type, the model tends to apply generic standards of "aesthetics, clarity, information completeness," and struggles to emphasize summarization for graphical abstracts or logical flow for technical roadmaps as human experts would. Additionally, VLMs have limited sensitivity to fine-grained visual details (such as axis labels and arrow directions), easily producing judgments inconsistent with humans. The reason SciDiagBench needs "preference guidance" is precisely to compensate for VLMs' tendency toward criteria drift under unconstrained conditions.

SciDiagBench's Solution: Categorization and Preference Guidance

The core contribution of SciDiagBench lies in constructing a systematic benchmark dataset and developing a human preference-guided evaluation method. The dataset divides scientific figures into 11 types, covering 10 broad domains, with a total of 165 scientific figures. These types include graphical abstracts, technical roadmaps, and illustrative examples.

In terms of evaluation methodology, SciDiagBench employs a preference-guided calibration process that incorporates human preferences into the generation of type-specific criteria. Simply put, the system does not directly apply preset rules but first learns human preference patterns when evaluating different figure types, then generates customized evaluation criteria for each figure type accordingly. This design enables automatic evaluation to better reflect the judging logic of human experts.

Human preference-guided calibration is a method that transforms human pairwise comparisons or rating data into evaluation criteria. Its typical workflow is: first collect experts' multi-dimensional preference judgments on different figures (e.g., "which figure is better and why"), then extract actionable scoring dimensions and weights through preference learning (such as the Bradley-Terry model or directly prompting large models to summarize preference patterns), and finally instantiate these dimensions as checklists tailored to specific figure types. Unlike directly fine-tuning evaluation models, the calibration process does not change VLM parameters but adjusts their "judging framework," making it lower-cost and more interpretable. SciDiagBench applies this idea to generate customized criteria for each of the 11 figure types, enabling the same VLM to switch evaluative perspectives across types, thereby reducing deviation from human experts.

Value and Limitations from Multiple Perspectives

According to the information provided in the paper, SciDiagBench's value lies in being the first to combine two dimensions—"type-specific" and "human preference alignment"—into automatic evaluation of scientific figures. Compared to generic benchmarks, it better captures the unique requirements of different figure types. However, publicly available details are currently limited—for example, the specific basis for dividing the 11 types, the concrete implementation of preference calibration, and the distribution of 165 figures across 10 domains are not elaborated in the original abstract.

Furthermore, the benchmark currently focuses mainly on evaluating the quality of scientific figure generation and has not yet addressed figure understanding or interactive scenarios. For academic publishing platforms or preprint services that need automated figure quality screening, SciDiagBench offers a more refined evaluation approach, but its practical deployment effectiveness still awaits further experimental validation.

A graphical abstract is a single schematic diagram on a paper's first page or beside the abstract, required to convey the core findings and innovations of the research at a glance, so evaluation focuses more on visual impact, information density, and conclusion readability. A technical roadmap is used to display research steps, stage dependencies, and decision branches, with evaluation focusing on logical coherence of the process, clarity of stage division, and semantic consistency between arrows and nodes. An illustrative example is typically used to explain methods or data, with evaluation emphasizing detail fidelity, annotation completeness, and reproducibility. The visual structural differences among these three figure types are precisely the fundamental reason why "type-specific criteria" need to exist—using the same set of generic metrics to measure all three simultaneously will inevitably lead to systematic underestimation or overestimation of certain types.

Implications for Scientific Figure Generation and Review

The emergence of SciDiagBench reflects that scientific figure evaluation is moving from "one-size-fits-all" toward "type-specific refinement." For researchers and tool developers using AI to generate scientific figures, this means evaluation feedback will become more targeted—graphical abstract evaluation will focus on summarization and visual appeal, while technical roadmaps will focus on logical coherence.

At the same time, this work also suggests a direction: human preferences can serve as a calibratable signal for improving the reliability of automatic evaluation systems. In the future, if such benchmarks can be extended to more figure types and interdisciplinary domains, they may help drive quality standards for scientific figure generation tools toward unification.

For those interested in AI applications in scientific research workflows, SciDiagBench is a signal worth noting—automatic evaluation is learning to "speak according to type" like human experts.

One-line Summary

SciDiagBench achieves type-specific automatic evaluation of scientific figures through figure categorization and human preference calibration.

Paragraph Summary

SciDiagBench addresses the "one-size-fits-all" shortcomings of existing vision-language model evaluation benchmarks by proposing a type-specific and human preference-aligned scientific figure evaluation scheme. The benchmark systematically divides scientific figures into 11 types (such as graphical abstracts, technical roadmaps, and illustrative examples), covering 10 broad domains with a total of 165 figures. Methodologically, it employs a preference-guided calibration process that first learns human experts' preference patterns when evaluating different figure types, then generates customized evaluation criteria for each type, making automatic evaluation closer to human judgment. The article notes that this work is the first to combine the two dimensions of "type-specific" and "human preference alignment," providing a more refined approach for figure quality screening on academic publishing platforms, but the specific categorization basis, calibration implementation details, and practical deployment effectiveness still await further experimental validation.