Unverified50% confidenceFactExact time
MAPLE's probability-level metrics measure whether the ranking of the correct token (the gold token) shifts among failure cases that were not recovered; even if the answer is ultimately not corrected, a rise in the gold token's ranking indicates that the technique is moving closer to the correct answer.
1
Sources
50%
Confidence
Long-term
Relevance
10/15/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedRecording negative samples where 'a recommended solution did not apply' is just as important as recording successes; otherwise the model will fall into statistical bias, mistaking high-frequency solutions for high-accuracy ones71% similarUnverifiedBenchmark scores measure the average of a model's success consistency and failure frequency, not its peak capability.71% similarUnverifiedScoring with completion-style proxy metrics will classify failures that terminate cleanly but produce incorrect results as successes, which is a flaw in many current agent evaluations.70% similarUnverifiedIn evaluation regression detection, only if the candidate's lower bound is higher than the baseline's lower bound (or the baseline point estimate) is there statistical grounds to say it has not gotten worse; otherwise it can only be judged inconclusive.70% similarUnverifiedThe core property of a strictly proper scoring rule is that the expected score is maximized if and only if the forecaster reports the probability they truly believe, with common examples including log-likelihood loss and the Brier score.69% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1038560API
curl https://kongchang.com/api/v1/knowledge/claims/1038560MCP
get_claim(id=1038560)