More Accurate Confidence, Yet Knowledge Graph Auditing Remains Hard to Improve?

In the pipeline where LLMs construct knowledge graphs, confidence estimation is often used to guide manual review. Intuitively, more accurate confidence should directly improve auditing efficiency. But a study points out that this improvement may be far smaller than expected—even if error predictions become more accurate, auditing effectiveness may not significantly improve. Through three concepts—repairability, selection margin, and label validity—the study reveals the gap between confidence and auditing benefit.

Why Auditing Does Not Improve Linearly with Confidence

The study models non-adaptive auditing as a budget-constrained coverage problem and characterizes the minimal error statistics relevant to decision-making. The core finding is: the auditing gain brought by improved confidence is limited by the structural distribution of errors. That is, if errors are concentrated in samples that are difficult to repair or whose labels are themselves questionable, even more accurate confidence is hard to translate into improved auditing effectiveness.

The authors further approximate the auditing strategy with "scalar-preserving first-order planning" and bound its regret with structural quality Δ^cl. This explains why a simple error-rate × fan-out queue and a joint model, under the same actions, can differ by 3.98–4.74 percentage points in residual errors (on held-out cross-document graphs).

Non-adaptive auditing means the auditing strategy is determined once and for all after obtaining confidence scores, without dynamically adjusting subsequent choices based on new information discovered during auditing. In contrast, adaptive auditing updates its estimate of remaining errors after each inspection, thereby potentially adjusting the subsequent auditing order. Non-adaptive auditing is easier to implement and evaluate, but may fail to fully exploit information gradually revealed during auditing.

The budget-constrained coverage problem is a classic problem in combinatorial optimization: given a total budget (e.g., how many facts can be checked at most), select a set of actions to maximize the number of covered errors or minimize residual errors. Here, each candidate fact corresponds to a possible repair action, and the auditor must select a subset of actions within a limited budget so that the final residual errors are minimized. This modeling treats auditing as a variant of set cover, allowing the use of approximation algorithms and regret-bound analysis tools.

Experimental Design and Key Data

The study simulates ideal repairs on Re-DocRED, targeting five extraction configurations and two cross-document graphs, then freezes confirmation on the test set and reproduces on 500 hash-selected training documents. Results show:

• On held-out cross-document graphs, the error-rate × fan-out queue leaves 3.98–4.74 percentage points more errors than the joint model.
• After opening shared actions to the joint planner, residual errors decrease by 0.89–1.43 percentage points in point estimates relative to its fact-only plan.
• The equivalence between the joint model and the independent-block model remains undetermined.

These numbers indicate that the structure of the auditing strategy (whether shared actions are allowed) affects the final error rate more than merely improving confidence accuracy.

Re-DocRED is a benchmark dataset for document-level relation extraction, extended from DocRED, with some annotation errors in the original dataset corrected through manual review, providing more reliable entity and relation labels. This dataset is commonly used to evaluate document-level information extraction models, and is especially suitable for studying annotation noise and auditing strategies because its corrected labels can serve as a relatively clean "gold standard" to simulate the repair process.

Cross-document graphs refer to global knowledge graphs formed by connecting entities and relations across multiple documents through coreference resolution or entity linking. Compared with single-document graphs, cross-document graphs involve more complex entity disambiguation and relation reasoning, have a wider range of error propagation, and auditing actions may require cross-document coordination. This paper evaluates auditing strategies on cross-document graphs precisely to examine the impact of the structural distribution of errors on auditing benefit.

The Role of Repairability and Label Validity

The study emphasizes that auditing benefit depends on whether errors are repairable and whether labels are valid. If errors stem from systematic bias in the extraction configuration, or if the benchmark labels themselves have omissions, then no matter how accurate the confidence is, it cannot guide auditors to the truly repairable errors. The authors mention that general LLM verification improved held-out set performance in three of four model-budget configurations, and LLM adjudication also identified possible benchmark omissions—suggesting that some "errors" may actually be label issues.

This perspective extends auditing from a "ranking problem" to a "structural problem": both the repairability of errors and the reliability of labels need to be considered.

First-order planning here refers to an optimization method that approximates the auditing strategy: modeling the selection of auditing actions as a first-order (linear) planning problem and simplifying the solution by preserving a scalar objective (such as the number of residual errors). The authors approximate the original problem with "scalar-preserving first-order planning" in order to derive a regret bound for the auditing strategy under computable conditions—that is, an upper bound on the performance gap between this approximate strategy and the optimal strategy. The regret bound is characterized by structural quality Δ^cl, reflecting how the structure of error distribution (such as repairability and label validity) limits the optimality of the approximate strategy. This analytical tool explains why merely improving confidence accuracy cannot break through structural limitations.

Implications for Knowledge Graph Auditing Practice

For teams building knowledge graphs, this study suggests: do not blindly pursue accuracy improvements in confidence models, but focus on the design of auditing actions. For example, allowing the joint planner to share actions (such as correcting entities and relations simultaneously) reduces residual errors more than merely optimizing confidence ranking. In addition, introducing LLM verification and adjudication can help identify benchmark omissions, but attention must be paid to differences in their effectiveness across configurations.

The study also leaves open questions: the equivalence between the joint model and the independent-block model has not been concluded, and auditing strategies for cross-document graphs still require further exploration.

In summary, better confidence is a necessary but not sufficient condition. The structural design of the auditing system, error repairability, and label quality jointly determine the final benefit.