LLMs Meet Knowledge Graphs: A Benchmark for Multi-hop Reasoning and Hallucination Suppression

Large language models are increasingly used in biomedicine and cognitive neuroscience, but whether they can reason reliably over heterogeneous knowledge graphs remains an open question. A newly released benchmark study systematically evaluates five mainstream models across five knowledge graphs and delivers a counterintuitive conclusion: domain-specific models are not necessarily better than general-purpose models at specialized tasks.

Evaluation Design: Five Models, Five Knowledge Graphs

The study selected five models—GPT-5.1, Gemini 2.0 Pro, Llama 3, Qwen, and BioGPT—covering both general-purpose LLMs and biomedical-specific models. On the knowledge graph side, it covers five heterogeneous graphs: DrugBank, NeuroSynth, Cognitive Atlas, Brain Map, and GDPKG, spanning domains such as drugs, neuroimaging, and cognitive science.

To measure reasoning depth and factual accuracy, the team built a multi-level query dataset and compared performance under two conditions: KG-aware (graph-grounded, i.e., the model can leverage knowledge graph structural information) and KG-agnostic (no graph structure information provided). For annotation consistency, Cohen's kappa was 0.781 (range 0.753–0.807) and Fleiss' kappa was 0.77—a relatively high level of agreement that provides a foundation for the credibility of the evaluation results.

Knowledge graphs and heterogeneous graphs: A knowledge graph (KG) organizes facts as "entity–relation–entity" triples, such as "aspirin—treats—headache." In engineering practice, it is typically represented as a typed directed graph, where nodes are concepts such as drugs, brain regions, and cognitive functions, and edges are the semantic relations between them. A so-called "heterogeneous graph" means the graph contains multiple types of nodes and edges—for example, DrugBank includes both drugs and target proteins, while Cognitive Atlas includes both cognitive tasks and psychological constructs. Heterogeneity means the model cannot rely on a single relation pattern alone but must understand differences among ontologies. This is precisely where multi-hop reasoning becomes difficult: a path may cross multiple semantic levels such as drugs, genes, and brain regions, and the model must select the correct relation type at each step.

KG-aware vs. KG-agnostic: These two conditions correspond to whether the model can "see" the graph structure. In the KG-agnostic setting, the model receives only a natural-language description of the question and must answer from pretrained memory; in the KG-aware setting, the model can access adjacency relations or subgraphs from the graph, effectively injecting structured retrieval results as context. The performance gap between the two directly quantifies the contribution of graph-structural information to factual accuracy and is the basic experimental design the study uses to assess the value of "graph grounding."

Key Finding 1: General-Purpose Models Excel at Higher-Order Reasoning

The results show that the general-purpose models GPT-5.1 and Gemini 2.0 Pro generally outperform domain-specific models on multi-hop and higher-level reasoning tasks. This advantage persists as query complexity increases, although all models' performance declines with greater complexity.

This finding challenges the intuition that "specialized models are always stronger." The study explicitly states that, contrary to the initial hypothesis, domain-specific models did not consistently surpass general-purpose models on specialized tasks under the no-graph-grounding condition. This suggests that the broad language capabilities accumulated through large-scale pretraining of general models may be more critical in complex reasoning scenarios than the gains from domain fine-tuning.

Key Finding 2: Graph Grounding Reduces Hallucination

Another important conclusion is that KG grounding reduced hallucination rates across all evaluated models. When models can leverage structured information from knowledge graphs, factual errors decrease markedly. This provides empirical support for "neural-symbolic hybrid architectures"—combining the language capabilities of large models with structured domain knowledge is a viable path to suppressing hallucination.

However, graph grounding is not a panacea. The study emphasizes that model performance still declines as query complexity increases, indicating that simply introducing graph structure cannot fully compensate for the ceiling on reasoning ability.

Hallucination and neural-symbolic hybrid architectures: Hallucination refers to a model generating content that appears plausible but is factually incorrect; in biomedical settings, this may manifest as fabricating drug interactions or misattributing brain-region functions. Graph grounding suppresses hallucination because it shifts part of the factual judgment from the model's parametric memory to external structured knowledge, reducing the room for the model to "answer from impression."

Neural-symbolic architectures go a step further: neural networks handle language understanding and flexible reasoning, while symbolic systems (such as knowledge graphs and rule engines) handle factual constraints and verifiability, with the two coupled via retrieval, tool calling, or constrained decoding. The core advantage of such architectures is interpretability—every conclusion can be traced back to a specific path in the graph. But the study also points to their boundary: symbolic knowledge can only constrain "what the model knows," not break through "how deeply the model can reason," so performance degradation under increasing complexity remains.

Practical Implications: How to Choose an LLM-KG Integration Strategy

Based on these results, the study offers practical guidance for selecting an LLM-KG integration strategy according to task complexity and computational constraints. For complex tasks requiring multi-hop reasoning, a general-purpose large model combined with graph grounding is the safer combination; in scenarios with limited computational resources, the additional overhead introduced by graph retrieval must be weighed.

The study also discusses the application prospects of hybrid neural-symbolic architectures in areas such as personalized medicine, cognitive impairment diagnosis, and explainable AI. What these scenarios share is a need for both the language understanding capabilities of language models and extremely high requirements for factual accuracy—exactly the value that graph grounding can provide.

Limitations and Open Questions

The value of this study lies in providing a systematic cross-model, cross-graph comparison framework, but the coverage of five knowledge graphs remains limited, and more analysis is needed on how structural differences among heterogeneous graphs affect reasoning performance. In addition, the conditions under which domain-specific models can truly leverage their advantages remains an open question.

For engineering practitioners, the most direct takeaway may be: do not assume domain models are stronger by default, and instead incorporate graph grounding into architecture design as a standard means of reducing hallucination.

One-line Summary

A new benchmark shows general-purpose LLMs outperform domain-specific models on higher-order knowledge graph reasoning, and graph grounding reduces hallucination.

Paragraph Summary

This study systematically evaluates five models—GPT-5.1, Gemini 2.0 Pro, Llama 3, Qwen, and BioGPT—on reasoning performance across five heterogeneous knowledge graphs: DrugBank, NeuroSynth, Cognitive Atlas, Brain Map, and GDPKG, comparing KG-aware and KG-agnostic conditions. The results yield two counterintuitive conclusions: first, general-purpose models generally outperform domain-specific models on multi-hop and higher-order reasoning tasks, and domain models do not consistently dominate without graph grounding; second, graph grounding reduces hallucination rates across all models, providing empirical support for neural-symbolic hybrid architectures. The study also notes that all models' performance declines as query complexity increases, and graph grounding cannot compensate for reasoning ceilings. Based on this, the authors provide practical guidance for choosing LLM-KG integration strategies according to task complexity and computational constraints, and discuss application prospects in personalized medicine and explainable AI.