DeRef
A method for validating arguments in LLM tool calls. An independent model derives the correct reference text from the full dialogue context, decoupling derivation from judgment and helping judge models assess the correctness of tool-call argument values more accurately.
Timeline (last 90 days)
The researchers proposed the DeRef (Derived Reference) scheme: an independent model reads the full conversation history (but does not look at the candidate value) to generate reference text for the judge, and the judge then makes its judgment based on the reference text.
DeRef's design decouples the two steps of "derivation" and "judging": the independent model is responsible for deriving the reference value, and the judge only needs to do the comparison work.
On three benchmarks, DeRef improved the AUROC of parameter judgment for all judge models tested.
In the controlled source-migration experiment, DeRef raised AUROC from chance level or below to between 98.40 and 99.59.
Do not let the judge simultaneously take on the two responsibilities of "derivation" and "verification"; assigning the derivation step to a dedicated model and letting the judge focus on comparison can significantly improve the reliability of the entire evaluation pipeline.
All Facts (5)
The researchers proposed the DeRef (Derived Reference) scheme: an independent model reads the full conversation history (but does not look at the candidate value) to generate reference text for the judge, and the judge then makes its judgment based on the reference text.
50%UnverifiedDeRef's design decouples the two steps of "derivation" and "judging": the independent model is responsible for deriving the reference value, and the judge only needs to do the comparison work.
50%UnverifiedOn three benchmarks, DeRef improved the AUROC of parameter judgment for all judge models tested.
50%UnverifiedIn the controlled source-migration experiment, DeRef raised AUROC from chance level or below to between 98.40 and 99.59.
50%UnverifiedDo not let the judge simultaneously take on the two responsibilities of "derivation" and "verification"; assigning the derivation step to a dedicated model and letting the judge focus on comparison can significantly improve the reliability of the entire evaluation pipeline.
50%