SciLitBench: A Full-Pipeline Benchmark for Evaluating LLM Capabilities in Systematic Literature Reviews

The Challenge of Bringing AI to Systematic Literature Reviews

Systematic Literature Reviews (SLRs) are among the most time-consuming tasks in academic research. SLRs originated from the evidence-based medicine movement, with standardized procedures established by the Cochrane Collaboration in the 1990s and later extended to fields such as social sciences and computer science. Unlike traditional narrative reviews, SLRs require researchers to pre-register their study protocols (e.g., on the PROSPERO platform), follow the PRISMA reporting guidelines, and ensure the reproducibility of search strategies and the transparency of screening processes. A typical SLR takes 6–18 months from initial search to final publication, with roughly 40% of that time spent on title-abstract screening and full-text assessment. Researchers often need to screen thousands of papers one by one, assess quality, and extract key data. Although large language models (LLMs) have demonstrated powerful capabilities in text comprehension and generation tasks, existing evaluation schemes tend to focus on only a single stage of the literature review process, lacking a systematic examination of the complete pipeline.

SciLitBench was designed precisely to fill this gap. As the first benchmark covering the full pipeline of literature reviews, it includes fine-grained annotations across 42,981 search records, 1,012 full-text papers, and 888 included studies. The benchmark aims to answer one core question: To what extent can LLMs actually handle systematic literature review tasks?

Three-Stage Evaluation: A Complete Performance Landscape from Screening to Extraction

SciLitBench is designed around the real-world literature review workflow, featuring three key evaluation stages, each with distinct capability requirements for LLMs.

Title and Abstract Screening: Structured Criteria Deliver Significant Improvements

The research team tested 22 open-source LLMs from six model families. Results showed that when researchers provide explicit inclusion and exclusion criteria, the F2 score improves by 28.8%, with significant gains in both recall and precision. The F2 score is a variant of the F-beta score family with beta set to 2, meaning recall is weighted four times more heavily than precision. The choice of F2 over the more common F1 score in the literature screening context reflects a core practical principle: the cost of missing a relevant paper (false negative) far outweighs the cost of including an irrelevant one (false positive), since overlooking key studies can introduce systematic bias into review conclusions.

This finding has significant practical implications: Structured screening rules are the key to unlocking LLMs' screening potential. Researchers using AI tools will see substantial returns from investing time in formulating clear screening criteria.

Full-Text Screening: Expert Rationales Effectively Guide Model Judgments

At the full-text screening stage, task complexity increases significantly—LLMs need to understand richer context and subtle nuances in academic discourse. Experimental data showed that researcher-written rationales improve model performance by approximately 15%. The reasoning processes of human experts effectively guide models toward more precise inclusion/exclusion decisions. These rationales are not simple keyword lists but logical explanations for why a paper should be included or excluded—for example, "Although this study involves the target intervention, the subjects are animal models rather than human participants, which does not meet the inclusion criteria." Injecting this type of chain reasoning is essentially a practical application of Chain-of-Thought prompting strategies in an academic setting.

Data Extraction: Clear Limits on Reliability

The data extraction stage reveals a strikingly different performance profile, with vast gaps between different types of extraction tasks:

• Structured information extraction: Models achieve an accuracy of 0.97 when extracting clearly formatted data such as publication year
• Methodological information extraction: Jaccard overlap drops sharply to 0.37 when identifying computational methods. Jaccard overlap measures the ratio of the intersection to the union of the model's extracted concept set and the gold standard set—0.37 means the model captures only about one-third of key methodological information
• Deep semantic extraction: Even the best-performing models recover only 30% of evaluation evidence and 25% of limitation descriptions

The reasons why LLMs excel at structured information extraction but struggle with deep semantic extraction span multiple levels. Structured data like publication years have highly consistent expression patterns in pretraining corpora, making it easy for models to develop stable extraction patterns. In contrast, descriptions of a paper's limitations are often scattered across different paragraphs of the discussion section, requiring cross-paragraph reasoning and synthesis. Furthermore, although current LLM context windows have expanded to 128K tokens or longer, information retrieval within long documents still suffers from the "Lost in the Middle" phenomenon—models pay significantly less attention to information in the middle of a document compared to the beginning and end. Complete extraction of evaluation evidence is fundamentally an open-ended information extraction task lacking clear boundary signals, which poses a fundamental challenge to current autoregressive LLM architectures.

This set of data clearly reveals the current capability ceiling of LLMs in academic information extraction.

Practical Boundaries of LLM-Assisted Literature Reviews

SciLitBench's most important contribution lies in delineating the practical boundaries of LLMs at each stage of the literature review process:

Tasks Suitable for LLMs: High-Recall Screening

LLMs perform reliably in both the title/abstract screening and full-text screening stages, especially when supported by clear criteria and rationales. These repetitive, high-recall-demand tasks are well-suited for independent or semi-automated LLM processing, significantly reducing the time researchers spend on initial screening.

Tasks Still Requiring Human Leadership: Complete Evidence Extraction

For tasks requiring deep understanding of paper content, detailed extraction of specific evidence, and analysis of limitations, LLM performance drops noticeably. This type of work still requires deep involvement from domain experts, with LLMs better suited as assistive tools rather than replacements.

Clearly defining this boundary holds significant reference value for both academic researchers selecting AI tools and developers optimizing product directions.

Implications for Academic Research and AI Tool Development

SciLitBench's value extends beyond the benchmark itself. It provides reproducible evaluation resources and standardized design principles, laying the groundwork for future research in related fields.

Implications for researchers: When using LLMs to assist with literature reviews, researchers should invest effort in formulating clear screening criteria and rationales, deploy LLMs at the screening stage to maximize efficiency gains, and maintain manual review during the data extraction phase. Specifically, structuring screening criteria using templates like the PICO framework (Population, Intervention, Comparison, Outcome)—a standard in academic domains—and embedding them into prompts yields better model judgment accuracy than simple natural language instructions.

Implications for tool developers: The role of Prompt Engineering in academic settings should not be underestimated. Prompt engineering refers to the technique of carefully designing input prompts to guide LLMs toward desired outputs, and it faces unique challenges in academic literature review contexts—academic texts are filled with domain-specific terminology, implicit assumptions, and complex chains of logical reasoning. SciLitBench's experiments demonstrate that explicit criteria and expert reasoning logic can significantly enhance model performance, presenting a stark contrast to prompt engineering in general-purpose settings and providing an empirical foundation for designing more effective human-AI collaboration interfaces.

Outlook for future directions: As LLM capabilities continue to advance, the reliability boundary for evidence extraction is expected to gradually expand. Technical approaches such as Retrieval-Augmented Generation (RAG), long-context modeling optimization, and domain-specific fine-tuning for academic scenarios may all prove to be key directions for breaking through current performance bottlenecks. The multi-stage evaluation framework established by SciLitBench will continue to serve as an objective yardstick for progress in this field.

Key Takeaways