The Derivation Gap in LLM Tool-Call Judging: How DeRef Fixes It

When LLM agents execute tasks, they frequently need to call external tools. Whether the argument values for a tool call are correct is often checked by a "judge model." But a new study finds that these judges have a systematic flaw: when the input contains no "source" for a value, the judge is more likely to reject it even when the value is completely correct. The researchers named this phenomenon the "derivation gap."

What Is the Derivation Gap

A tool-call judge typically works like this: given the conversation history, determine whether the argument values proposed by the LLM agent are correct. Ideally, the judge should base its decision on an understanding of the function schema, the conversation context, and the task goal. But in practice, that's not what happens.

The study found that when the judge's input lacks direct "reference text" for a value—that is, there is no passage explicitly stating what the value should be—the judge's probability of accepting a correct value drops significantly. This occurs in three scenarios: the value comes from the function schema's default but was never mentioned by the user; the value must be inferred from context; or the text containing the value is truncated outside the judge's input.

This drop is defined as the "derivation gap." It appears across all seven judge models the researchers evaluated, including the System One series CLM and Jev.

Function schema is a foundational concept in tool-calling mechanisms. In LLM agent scenarios, each external tool (such as search, calculation, or database query) must declare its interface definition to the model, and that definition is the function schema. It typically uses JSON Schema format to describe the tool's name, purpose, and each parameter's name, data type, required status, and default value. For example, a weather query tool might declare location as a required string and unit as an optional string with a default value of celsius. When the agent decides to call a tool, it must generate an argument object that conforms to that schema. When the judge model determines whether arguments are correct, the schema itself is also one of the important bases—because default values often exist only in the schema and are never explicitly mentioned by the user or the agent in the conversation history. This is exactly the scenario where the derivation gap is most likely to occur: the value is correct, but the only textual source "supporting" it is somewhere the judge cannot see.

It's Not Just a Truncation Problem

A natural hypothesis is that the derivation gap is merely an artifact of input truncation—information was cut off, so of course the judge can't determine correctness. But the experimental data refutes this explanation.

In a benchmark with no truncation at all, correct default values actually exhibited the largest derivation gap. This means the problem is not information loss, but that judges fundamentally rely on explicit reference text in the input rather than independently verifying a value's correctness.

The researchers designed a controlled "source migration" experiment to verify this. The results were very direct: judges tend to accept an incorrect value that is visible in the input while rejecting a correct value whose source is hidden. In other words, judges "follow the reference"—their basis for judgment is not whether the value itself is right, but whether there is text in the input supporting that value.

AUROC (Area Under the Receiver Operating Characteristic curve) is a common metric for evaluating the performance of binary classification models. The ROC curve plots the false positive rate (FPR) on the x-axis and the true positive rate (TPR) on the y-axis, showing performance across different decision thresholds; the area under the curve is AUROC, ranging from 0 to 1. 0.5 is equivalent to random guessing, and 1.0 represents perfect discrimination. In the judge model setting, the positive class is "argument value correct" and the negative class is "argument value incorrect," so AUROC measures the probability that the judge scores a correct value higher than an incorrect one. The advantage of using AUROC over simple accuracy is that it is unaffected by the decision threshold and the ratio of positive to negative samples, making it convenient for cross-benchmark and cross-model comparison. Therefore, in this article, improving AUROC from random-level to above 98 shows that the judge can almost completely separate correct values from incorrect ones, which is a very significant improvement.

DeRef: Let an Independent Model Generate the Reference

To address this problem, the researchers proposed DeRef (Derived Reference). The core idea is simple: use an independent model to read the full conversation history (but not the candidate values), then write a "reference text" for the judge explaining what each argument value should be. The judge then makes its decision based on this reference text.

This design decouples the two steps of "derivation" and "judging." The independent model is responsible for deriving the correct reference values from context, while the judge only needs to perform comparison work rather than deriving everything from scratch itself.

The Effect of DeRef

On three benchmarks, DeRef improved the AUROC of argument judgment (area under the curve, a metric measuring binary classification ability) for all judge models tested.

Even more striking are the results of the controlled experiment: in the source migration test, DeRef raised AUROC from random level or lower to between 98.40 and 99.59. This shows that once judges receive reliable reference text, their judgment ability approaches perfection.

Implications for Tool-Call Evaluation

This study reveals an easily overlooked problem: when we evaluate an agent's tool-calling ability, the reliability of the judge model directly determines the credibility of the evaluation conclusions. If judges systematically favor values that are "documented" and reject correct values that "require reasoning," then correct arguments that require complex reasoning may be incorrectly marked as failures during evaluation.

The value of the DeRef approach is not only that it improves judge accuracy, but also that it points to a general direction for improvement—do not make the judge simultaneously bear the two responsibilities of "derivation" and "verification." Assigning the derivation step to a dedicated model and letting the judge focus on comparison can significantly improve the reliability of the entire evaluation pipeline.

One-line Summary

The study finds a "derivation gap" in tool-call judges and proposes DeRef to decouple derivation from judging as a fix.

Paragraph Summary

This article introduces a new study on LLM agent tool-call judges. The researchers found that when the judge's input lacks explicit reference text for an argument value, the probability of the judge accepting it drops significantly even if the value is completely correct, a phenomenon they named the "derivation gap." This flaw appears across all seven judge models evaluated, and it is not caused by input truncation—in a no-truncation benchmark, correct values from schema defaults actually exhibited the largest gap. A controlled "source migration" experiment further proved that judges tend to accept incorrect values visible in the input while rejecting correct values whose source is hidden, showing that they are "following the reference" rather than independently verifying. To address this, the researchers proposed DeRef: use an independent model to generate reference text from the full conversation history, and let the judge simply compare. DeRef improved AUROC for all judge models on three benchmarks, reaching as high as 98.40–99.59 in the controlled experiment.