How Can Human-AI Complementarity Be Evaluated Without Ground Truth? A New Framework Proposes Adequacy Labels

Research proposes replacing ground truth with adequacy labels to extend human-AI complementarity evaluation to non-prediction tasks.
The article discusses the RSS-sourced 'Complementarity without ground truth?': human-AI complementarity is typically quantified via ground truth labels, but real interactions like medical advice concern adequacy relative to practical goals, not predicting an objective fact. Researchers propose context-dependent adequacy labels generated through a two-step scoring method inspired by medical multi-criteria decision analysis, explicitly revealing normative choices. Drawing on computational reliabilism, they identify technical, scientific, and social factors supporting label construction, making evaluation auditable. The framework extends quantitative complementarity research from prediction to decision-support scenarios, offering a methodological foundation where ground truth doesn't apply, though large-scale empirical validation is still lacking.
Human-AI Complementarity Doesn't Only Rely on Ground Truth
Human-AI complementarity refers to the ability of humans, with AI assistance, to perform beyond what either humans or AI can achieve alone. In prediction tasks, this improvement is typically evaluated using ground truth labels—labels produced by reference construction procedures designed to represent the true state of the situation being predicted. However, many real-world human-AI interactions are not prediction tasks. For example, when medical AI provides treatment recommendations, its output concerns adequacy relative to practical goals rather than predicting some objective fact.
According to the RSS-sourced article "Complementarity without ground truth?", evaluating outputs in such tasks involves multiple evaluation dimensions, and ground truth is not the appropriate reference standard. This raises a key question: when ground truth is absent, can we still quantitatively evaluate human-AI complementarity?

From Ground Truth to Adequacy Labels
The study examines in depth the dependence of complementarity on ground truth and attempts to extend it to human-AI interaction scenarios where ground truth does not apply. The authors demonstrate that tasks beyond prediction can still generate adequacy labels through reference construction procedures. To this end, they propose a method for extending complementarity to such settings: replacing ground truth labels with context-dependent adequacy labels.
The generation of these adequacy labels employs a two-step scoring method inspired by multi-criteria decision analysis frameworks in the medical domain. The method also explicitly reveals the normative choices underlying reference construction—the value judgments that are unavoidable in the evaluation process. Because the construction of adequacy labels embeds evaluative choices, the researchers draw methodological inspiration from computational reliabilism to identify the technical, scientific, and social factors that support label construction.

Multi-criteria decision analysis (MCDA) is a class of systematic methods for ranking or classifying alternatives under multiple, often conflicting evaluation criteria. A typical process includes: defining decision objectives and a criteria system, assigning weights to each criterion, and aggregating performance scores across criteria into a single utility value. Common aggregation methods include weighted sum, analytic hierarchy process (AHP), ELECTRE, and PROMETHEE. MCDA is widely adopted in medical decision-making, for example in resource allocation, treatment selection, and health technology assessment, where efficacy, safety, cost, and patient preferences are weighed. In this framework, MCDA's role is to synthesize multiple adequacy dimensions (such as safety, effectiveness, and acceptability) into a comparable label; its key premise is that weights and aggregation rules must be explicitly declared, otherwise the synthesized result will implicitly contain unstated value judgments.
Method Core: Two-Step Scoring and Normative Choices
The core of the framework lies in the two-step scoring method. The first step may involve defining task-relevant evaluation dimensions, and the second step synthesizes these dimensions into adequacy labels through multi-criteria decision analysis. This process is not value-neutral—it requires researchers to make explicit normative choices, such as which dimensions matter, how to weight them, and what state counts as "adequate."
Computational reliabilism provides methodological support for these choices. It emphasizes that the label construction process must attend to technical reliability (such as the reliability of scoring instruments), scientific reliability (such as the evidence base for dimension selection), and social reliability (such as stakeholder acceptance). By explicitly identifying these factors, researchers can more transparently justify the reasonableness of adequacy labels, thereby introducing auditable quantitative standards into human-AI complementarity evaluation.
Computational reliabilism borrows the core intuition of reliabilism in epistemology: whether a belief or judgment has epistemic justification depends on whether the process producing it is reliable, rather than on whether it corresponds to some ultimate truth. Transposed to the computational evaluation context, this means the justification of adequacy labels should be traced back to the entire process that generates them—whether scoring instruments are stable, whether dimension selection is evidence-supported, and whether construction procedures can withstand scrutiny by the relevant community. This stance is particularly suited to scenarios where ground truth is absent, because it shifts the evaluation focus from "whether the label corresponds to objective fact" to "whether the label is produced by a trustworthy procedure." Researchers can thus treat technical reliability, scientific evidence, and social acceptance as separately reviewable sources of reliability, making normative choices discussable and accountable.
Expanding the Empirical Research Boundaries of Human-AI Complementarity
The significance of this work lies in extending empirical research on human-AI complementarity from prediction tasks to decision-support scenarios. In fields such as healthcare, policy, and management, the outputs of AI systems often cannot be measured with simple right-or-wrong labels but instead require judgments of adequacy in specific contexts. The new framework makes it possible to quantitatively evaluate human-AI complementarity in these scenarios, providing a methodological foundation for designing more effective human-AI collaborative systems.
Although the study remains at the methodological level and has not yet reported large-scale empirical results, it opens the way for future work: how to construct reliable adequacy labels across different domains, how to validate the validity of these labels, and how to compare complementarity conclusions under different reference construction procedures. These questions deserve continued attention from researchers in human-computer interaction and AI evaluation.
One-line Summary
Research proposes replacing ground truth with adequacy labels to extend human-AI complementarity evaluation to non-prediction tasks.
Paragraph Summary
The article discusses the core argument of the RSS-sourced "Complementarity without ground truth?": human-AI complementarity is typically quantified using ground truth labels, but real interactions such as medical recommendations concern adequacy relative to practical goals rather than predicting an objective fact. The researchers propose replacing ground truth with context-dependent adequacy labels, generated through a two-step scoring method inspired by medical multi-criteria decision analysis, while explicitly revealing the normative choices in reference construction. Drawing on computational reliabilism, they identify the technical, scientific, and social factors supporting label construction, making the evaluation process auditable. The framework extends quantitative research on complementarity from prediction tasks to decision-support scenarios, providing a methodological foundation for domains where ground truth does not apply, though large-scale empirical validation is still lacking.
Related articles

AI Writes Your Code—Do You Still Need to Learn Git? Five Reasons
AI coding agents handle Git operations automatically—is it still necessary to learn Git? Five reasons: team communication, reviewing agent PRs, monitoring code changes, fallback when AI goes down, and the joy of learning—plus how the focus of learning Git is shifting in the agent era.

Anthropic Reportedly Prices IPO at $83 Per Share, Valuing Company at $1.5 Trillion
Anthropic reportedly prices IPO at $83 per share, above the expected range, with a fully diluted valuation of $1.5 trillion. This article analyzes the signal significance of the pricing and its impact on the AI industry's financing environment.

Open-Source Code Hosting Platform Codeberg Suffers Outage, Sparking Debate in Developer Community
Open-source code hosting platform Codeberg.org suffered an outage, sparking Hacker News debate over nonprofit platform stability and open-source infrastructure dependency risks.