GAUGE
A research study published on arXiv that proposes a reusable offline evaluation protocol for assessing the ranking validity and construct validity of evaluation pipelines that combine LLM-simulated user conversations with LLM-judge scoring. The study reveals a systematic decoupling between user satisfaction and task success.
Timeline (last 90 days)
GAUGE is a reusable offline evaluation protocol proposed by researchers to test whether the ranking results of a 'simulator dialogue + LLM judge scoring' gate match well-documented and verifiable reward signals.
The GAUGE study covered 25 agents from six providers and tested them on two benchmarks: τ²-bench and SimulatorArena.
GAUGE distinguishes between two types of evaluation validity: ranking validity (whether stronger agents are correctly ranked higher) and construct validity (whether the measured metrics truly correspond to task success).
The study shows that user satisfaction carries almost no information about whether a task was truly completed, and conversations judged as 'satisfactory' are decorrelated from actual task success.
Among conversations rated as 'satisfactory' by blind evaluation panels, as many as 57.5% had not actually completed the customer's task.
The satisfaction–success gap pattern holds consistently across five different rater groups, two benchmarks, and all subjective evaluation dimensions.
For agent pairs with large reward gaps, the decision disagreement rate is below 1%.
For closely matched agent pairs, the decision disagreement rate surges to 31%.
GAUGE proposes a 'calibrate-then-trust' approach, arguing that subjective satisfaction signals can be retained but must be calibrated with verifiable objective success signals.
GAUGE recommends introducing a judge-free 'completion bit' as a zero-cost tripwire to capture performance regressions caused by truncated conversations.
All Facts (10)
GAUGE is a reusable offline evaluation protocol proposed by researchers to test whether the ranking results of a 'simulator dialogue + LLM judge scoring' gate match well-documented and verifiable reward signals.
50%UnverifiedThe GAUGE study covered 25 agents from six providers and tested them on two benchmarks: τ²-bench and SimulatorArena.
50%UnverifiedGAUGE distinguishes between two types of evaluation validity: ranking validity (whether stronger agents are correctly ranked higher) and construct validity (whether the measured metrics truly correspond to task success).
50%UnverifiedThe study shows that user satisfaction carries almost no information about whether a task was truly completed, and conversations judged as 'satisfactory' are decorrelated from actual task success.
50%UnverifiedAmong conversations rated as 'satisfactory' by blind evaluation panels, as many as 57.5% had not actually completed the customer's task.
50%UnverifiedThe satisfaction–success gap pattern holds consistently across five different rater groups, two benchmarks, and all subjective evaluation dimensions.
50%UnverifiedFor agent pairs with large reward gaps, the decision disagreement rate is below 1%.
50%UnverifiedFor closely matched agent pairs, the decision disagreement rate surges to 31%.
50%UnverifiedGAUGE proposes a 'calibrate-then-trust' approach, arguing that subjective satisfaction signals can be retained but must be calibrated with verifiable objective success signals.
50%UnverifiedGAUGE recommends introducing a judge-free 'completion bit' as a zero-cost tripwire to capture performance regressions caused by truncated conversations.
50%