Do LLMs Really Understand Context? A Knowledge Graph Evaluation Framework Offers New Answers

A Core Question That's Been Avoided

The linguistic capabilities demonstrated by large language models (LLMs) are nothing short of remarkable — yet a fundamental question continues to linger: do these models truly understand context, or are they simply performing sophisticated pattern matching at an unprecedented scale?

This isn't mere semantics. In question-answering (QA) tasks, a response may read as perfectly fluent yet bear no logical connection to the provided context — it's a pattern the model memorized from training data, not genuine reasoning grounded in the current situation. Distinguishing "understanding" from "memorization" is precisely the problem this research sets out to solve.

The researchers define LLM contextual understanding as a three-stage capability: correctly extracting relevant information from a given context, integrating it into a coherent internal representation, and then reasoning from that representation to produce factually consistent and contextually grounded responses. All three stages are essential — and existing evaluation methods consistently fail to see past the surface.

The Blind Spots of Traditional Evaluation Metrics

For years, metrics like BLEU (Bilingual Evaluation Understudy) and perplexity have dominated model evaluation. But these methods only measure surface-level performance — BLEU focuses on word overlap, while perplexity measures how confidently a model predicts a piece of text. Neither can determine whether a response is semantically and logically grounded in the context.

This exposes a critical gap in QA evaluation: a response with a high BLEU score might simply regurgitate seemingly relevant phrases while being factually indefensible. In other words, existing metrics cannot distinguish between "got it right" and "got lucky."

For practitioners building reliable AI systems, this evaluation blind spot means we may be systematically overestimating models' true comprehension — and we have no way to pinpoint exactly which step in the reasoning chain broke down.

BLEU was originally designed for machine translation, scoring outputs by counting n-gram (contiguous word sequence) overlaps between the model's output and a reference translation. Perplexity measures a language model's prediction uncertainty on test text — the lower the score, the more confidently the model predicts that text. The shared limitation of both metrics is that they are "form-driven": they observe only the surface of words, with no regard for semantic truthfulness or logical consistency. In a QA setting, a model can generate a fluent response with high n-gram overlap and low perplexity while seamlessly packaging misattributed facts or fabricated relationships. In recent years, embedding-based metrics like BERTScore have attempted to introduce semantic similarity, but they still operate at the sentence level and remain ill-equipped to track whether specific factual claims in an answer are consistent with the context.

S3KG: A Unified Measure of Structure and Semantics

To fill this gap, the research team proposed an evaluation framework built on knowledge graphs (KGs). At its core is a hybrid similarity metric called S3KG (Semantic Structural Similarity for KGs).

The elegance of S3KG lies in its ability to capture two types of signals simultaneously:

• Structural signals: Knowledge graphs organize information as triplets (i.e., "entity — relation — entity"). Structural similarity measures whether the knowledge structure of the model's output aligns with the reference graph.
• Semantic signals: Structural matching alone isn't enough. Semantic similarity ensures that entities and relations are also aligned at the level of meaning.

By fusing these two signals into a single score, S3KG provides a more comprehensive picture of whether a response has truly built a correct knowledge representation from the context — rather than relying on superficial lexical matching.

A knowledge graph (KG) is a graph-structured data format for storing real-world knowledge. Its basic unit is the "triplet": (head entity, relation, tail entity) — for example, (Einstein, born in, Germany). This structure offers two natural advantages: first, it makes implicit propositions in natural language explicit and computationally verifiable; second, the graph structure preserves multi-hop relationships between entities, enabling it to represent reasoning chains far more complex than any single sentence. The core idea behind S3KG is to convert both the reference answer and the model-generated answer into knowledge graphs, then compare them at the graph level across both structural topology (which triplets exist and how they connect) and semantic content (whether the meanings of entities and relations are aligned). This "structure first, then compare" pipeline fundamentally sidesteps the noise introduced by natural language's expressive diversity, directing evaluation at the logical skeleton of the information itself.

Triplet-Level Error Diagnostics

Beyond scoring, the research also introduces a companion diagnostic analysis framework. It identifies and categorizes reasoning errors at the triplet level, enabling fine-grained analysis of model failure cases.

This is particularly valuable in practice. Traditional evaluation typically tells you that a model "got it wrong" without explaining where. Triplet-level diagnostics can pinpoint the exact failure: was it entity recognition, relational reasoning, or information integration that broke down? This interpretable error analysis gives model developers a concrete direction for improvement.

Empirical Performance Across Nine Benchmarks

On the experimental side, the research team validated S3KG across nine benchmark tests. Compared to the strongest baseline methods, S3KG achieved F1 score improvements of up to 7.6 points, with an AUROC (Area Under the Receiver Operating Characteristic Curve) as high as 0.973.

An AUROC approaching 0.973 means the framework achieves extremely high accuracy in discriminating between "contextually correct" and "incorrect" responses — a value that approaches the theoretical upper bound of discriminative power.

These quantitative gains indicate that using knowledge graphs as an evaluation intermediary genuinely captures real differences in LLM contextual understanding more sensitively than traditional surface-level metrics.

AUROC (Area Under the Receiver Operating Characteristic Curve) is a standard metric for measuring a binary classifier's discriminative ability, ranging from 0 to 1. A score of 0.5 is equivalent to random guessing; 1.0 represents perfect classification. In the context of an evaluation framework, AUROC measures whether the framework can reliably distinguish "responses that genuinely reflect contextual understanding" from "responses that are superficially fluent but are actually recalled from memory." An AUROC of 0.973 means that in nearly 97% of cases, S3KG correctly ranks a correct response above an incorrect one — a level of confidence that is exceptionally high in the field of natural language evaluation, and one that validates the discriminative power of knowledge graph structure in capturing semantic consistency.

Significance and Outlook

The value of this work extends beyond proposing a new metric. It represents a shift in how we think about LLM evaluation: from measuring "how much does the output look right" to measuring "did the model genuinely understand and reason."

Knowledge graphs, as structured carriers of knowledge, are naturally suited to serve as this evaluative bridge. By mapping model responses onto graph structures and simultaneously accounting for semantics, S3KG provides a practical, quantifiable tool for answering that fundamental question — are LLMs understanding, or merely remembering?

For researchers, this framework and its accompanying diagnostic capabilities have the potential to become a standard instrument for dissecting model reasoning weaknesses. For application developers, it points toward a more rigorous evaluation path for building trustworthy QA systems. That said, questions around the cost of constructing knowledge graphs and cross-domain generalization remain open for future research to address.