MASCAL
A confidence calibration method for multi-agent collaboration that jointly optimizes task performance and confidence calibration through reinforcement learning, enabling agents to output calibrated probabilities of correctness and pass them to downstream agents during handoffs, and, combined with a confidence-aware weighting mechanism, improving the reliability and interpretability of tasks such as multi-hop question answering.
Timeline (last 90 days)
The MASCAL method lets each agent jointly optimize task performance and confidence calibration during reinforcement learning, learning to generate calibrated correct probabilities for its answers.
At handoff, MASCAL passes the correct probability of the answer to the downstream agent, providing the downstream with a quantified basis for decision-making.
MASCAL introduces a confidence-aware weighting mechanism: during training it increases the learning weight on positions sensitive to upstream probabilities, enabling agents to learn more from decisions that actively use the signal.
The experiments cover three collaboration architectures, involve web retrieval and reasoning tasks, and the results show that MASCAL improves the performance of both types of agents in task accuracy and calibration quality.
MASCAL can generalize to deeper collaboration chains and different model scales, and does not depend on a specific architecture or model size.
Through intervention analysis, the research team confirmed that upstream confidence does indeed affect downstream decision behavior.
The learned final confidence supports selective answering and targeted re-checking: when confidence is high, output is more confident; when confidence is low, re-examination of specific steps is triggered.
All Facts (7)
The MASCAL method lets each agent jointly optimize task performance and confidence calibration during reinforcement learning, learning to generate calibrated correct probabilities for its answers.
50%UnverifiedAt handoff, MASCAL passes the correct probability of the answer to the downstream agent, providing the downstream with a quantified basis for decision-making.
50%UnverifiedMASCAL introduces a confidence-aware weighting mechanism: during training it increases the learning weight on positions sensitive to upstream probabilities, enabling agents to learn more from decisions that actively use the signal.
50%UnverifiedThe experiments cover three collaboration architectures, involve web retrieval and reasoning tasks, and the results show that MASCAL improves the performance of both types of agents in task accuracy and calibration quality.
50%UnverifiedMASCAL can generalize to deeper collaboration chains and different model scales, and does not depend on a specific architecture or model size.
50%UnverifiedThrough intervention analysis, the research team confirmed that upstream confidence does indeed affect downstream decision behavior.
50%UnverifiedThe learned final confidence supports selective answering and targeted re-checking: when confidence is high, output is more confident; when confidence is low, re-examination of specific steps is triggered.
50%