SRBench: A Benchmark Framework for Automating Systematic Review Screening with LLMs

Systematic Reviews (SRs) are a cornerstone of evidence-based research, but the screening phase — determining which articles from a vast pool of literature are relevant to the research question — demands enormous time and effort from researchers. A newly published paper introduces SRBench, a benchmark framework specifically designed to address this pain point, along with a companion tool called PromptSR, with the goal of systematically evaluating the real-world performance of large language models (LLMs) in literature screening tasks.

Why Screening Is the Bottleneck in Systematic Reviews

In a standard systematic review, researchers may need to assess thousands — or even tens of thousands — of search results, evaluating each one against inclusion criteria. This work is repetitive, tedious, and highly prone to fatigue-induced errors, making it one of the most time-consuming steps in the entire review process.

LLMs are naturally well-suited for text understanding and classification tasks, which is why they've attracted significant attention as tools to assist with relevance judgments and dramatically reduce manual screening workload. The real challenge, however, is this: how do we objectively measure how well an LLM actually performs on this kind of task?

The paper points out that most existing evaluation approaches rely on traditional metrics like accuracy, which can be deeply misleading when applied to the highly class-imbalanced datasets typical of systematic review screening. The reason is straightforward — in real-world review data, the number of excluded articles far exceeds the number of included ones. A model that naively labels every article as "exclude" can still achieve a high accuracy score, yet it provides zero value to researchers.

Class imbalance is a common challenge in machine learning and NLP, referring to datasets where the number of samples per class differs dramatically. In systematic review screening, this problem is particularly extreme: out of the initial search results, only about 1–5% of articles are ultimately included, with the rest excluded. The traditional metric of "accuracy" fails badly in this setting — a "lazy model" that marks everything as "exclude" can achieve 95%+ accuracy while being completely useless. More appropriate metrics for this context include recall (how many truly relevant articles are correctly identified), precision, F1 score, and task-specific indicators like Work Saved over Sampling (WSS). The field broadly agrees that the cost of missing a key article (false negative) far outweighs the cost of mistakenly including an irrelevant one (false positive), so any evaluation framework must place recall at its core.

SRBench: A Benchmark Dataset with 45,000+ Labeled Entries

To address the lack of a unified evaluation standard, the paper constructs a benchmark dataset containing 45,064 labeled entries spanning 32 carefully curated secondary studies. This scale gives evaluation results cross-domain, cross-topic representativeness, rather than relying on scattered, one-off examples.

Equally important is the accompanying evaluation framework. It is specifically designed with class imbalance in mind, explicitly accounting for the natural overrepresentation of excluded articles relative to included ones in systematic reviews. In other words, rather than relying on traditional metrics that can be fooled by extreme class imbalance, it produces results that more accurately reflect a model's practical utility in real screening scenarios.

This design philosophy targets the core tension in deploying LLM-assisted screening: researchers' greatest concern is missing key articles that should have been included (false negatives), and the evaluation framework must genuinely capture a model's performance in this respect.

Secondary studies are research works that take already-published primary studies as their object of analysis, synthesizing and aggregating them to draw higher-order conclusions. Systematic reviews and meta-analyses are the most typical examples. Unlike primary studies that collect experimental data directly, the quality of secondary studies depends heavily on the rigor and comprehensiveness of the literature screening process — any important articles missed at the screening stage can bias all subsequent analyses. SRBench's choice to build its dataset on 32 carefully curated secondary studies ensures both academic rigor and broad coverage across different domains and inclusion criteria, making its benchmark results generalizable across fields rather than applicable only to a specific discipline or study type.

PromptSR: An Experimental Tool for Prompt Engineering

Beyond the data and framework, the paper also releases PromptSR, a tool designed to support LLM-based screening experiments. It provides three main capabilities:

• Prompt experimentation: Enables researchers to try different prompt designs and observe their effect on screening performance
• Experiment management: Systematically organizes and tracks multiple rounds of experiments
• Result analysis: Provides structured analysis of screening outputs

This reflects a pragmatic insight: in LLM-assisted tasks, the design of the prompt often has a decisive impact on the final outcome. A reproducible, manageable, and analyzable experimental tool allows researchers to move from "tweaking prompts by intuition" to "optimizing with evidence."

The paper also provides a worked example demonstrating how SRBench and PromptSR work together, giving future researchers a directly applicable practical reference.

Prompt engineering refers to the practice of carefully designing the text instructions fed to a large language model in order to guide it toward more desirable outputs, without modifying model weights. In literature screening, prompt design involves multiple dimensions: whether to include detailed descriptions of inclusion/exclusion criteria, whether to provide examples (few-shot), whether to ask the model to provide confidence scores or reasoning, and how strictly the output format is specified. Research has shown that different prompting strategies on the same model can lead to recall differences of more than 20 percentage points, making prompt engineering a critical variable in LLM screening performance. PromptSR brings this variable under systematic control, allowing researchers to compare different prompting strategies under controlled conditions — transforming "prompt tuning" from an intuition-driven trial-and-error process into a recordable, reproducible scientific experiment.

Implications for Evidence-Based Research Automation

The value of this work lies not in proposing a single high-performing model, but in filling a gap that the entire field has been missing — unified, trustworthy, and practically grounded evaluation infrastructure.

Previously, it was difficult to compare studies on LLM-assisted screening side by side: inconsistent metrics, limited dataset sizes, and ignored class imbalance made findings hard to generalize. SRBench provides a common yardstick; PromptSR provides a common experimental platform. Together, they have the potential to move the field from "everyone speaking their own language" to a standardized research paradigm built on reproducibility and comparability.

For teams looking to introduce AI assistance into their systematic review workflows, the emergence of benchmark frameworks like this means that selecting and fine-tuning models can finally be grounded in more reliable evidence — rather than being misled by impressive but distorted accuracy numbers.