Derivation Gap
A systematic deficiency of judge models for LLM tool calls: when the input lacks direct reference text for an argument value, the judge tends to reject even fully correct values. This indicates that judges rely on explicit reference text rather than independently verifying the correctness of values.
Timeline (last 90 days)
Tool-calling judge models have a systematic flaw: when no source for a value can be found in the input, the judge is more inclined to reject it even if the value is completely correct, a phenomenon named the "derivation gap".
The derivation gap appears in three scenarios: the value comes from a function schema default that the user never mentioned; the value must be inferred from context; or the text containing the value is truncated outside the judge's input.
In a benchmark with no truncation at all, correct default values instead exhibited the largest derivation gap, ruling out the explanation that truncation caused the artifact.
All Facts (3)
Tool-calling judge models have a systematic flaw: when no source for a value can be found in the input, the judge is more inclined to reject it even if the value is completely correct, a phenomenon named the "derivation gap".
50%UnverifiedThe derivation gap appears in three scenarios: the value comes from a function schema default that the user never mentioned; the value must be inferred from context; or the text containing the value is truncated outside the judge's input.
50%UnverifiedIn a benchmark with no truncation at all, correct default values instead exhibited the largest derivation gap, ruling out the explanation that truncation caused the artifact.
50%