MAPLE
A unified diagnostic framework for evaluating the impact of knowledge extraction enhancement (KEE) techniques on parametric knowledge. It decomposes changes in answer recovery, destruction and gold-standard ranking across three levels—surface, probability and beyond-sampling—to prevent aggregate accuracy from masking diverse effects.
Timeline (last 90 days)
MAPLE is a unified evaluation framework for characterizing how knowledge extraction enhancement (KEE) techniques affect parametric knowledge extraction, and its diagnostic metrics decompose the effects at three levels: the surface level, the probability level, and the beyond-sampling level.
MAPLE's surface-level metrics measure the recovery and breakage of answers, where recovery means a question originally answered incorrectly is answered correctly after KEE intervention, and breakage means a question originally answered correctly is instead answered incorrectly.
MAPLE's probability-level metrics measure whether the ranking of the correct token (the gold token) shifts among failure cases that were not recovered; even if the answer is ultimately not corrected, a rise in the gold token's ranking indicates that the technique is moving closer to the correct answer.
MAPLE's beyond-sampling level detects whether KEE techniques can bring answers missed by sampling under a fixed sampling budget back within reach, thereby distinguishing between two entirely different root causes of failure: missing knowledge and sampling luck.
MAPLE's value lies not in proposing new KEE techniques, but in providing a set of analytical lenses for understanding and comparing existing techniques, helping researchers and engineers see the composition of the "net benefit" and make scenario-specific trade-offs.
Against the backdrop of increasingly homogenized large-model evaluation, decomposing evaluation dimensions itself may bring new discoveries, and MAPLE's research suggests that asking what happens behind the score is more meaningful than chasing leaderboard rankings.
All Facts (6)
MAPLE is a unified evaluation framework for characterizing how knowledge extraction enhancement (KEE) techniques affect parametric knowledge extraction, and its diagnostic metrics decompose the effects at three levels: the surface level, the probability level, and the beyond-sampling level.
50%UnverifiedMAPLE's surface-level metrics measure the recovery and breakage of answers, where recovery means a question originally answered incorrectly is answered correctly after KEE intervention, and breakage means a question originally answered correctly is instead answered incorrectly.
50%UnverifiedMAPLE's probability-level metrics measure whether the ranking of the correct token (the gold token) shifts among failure cases that were not recovered; even if the answer is ultimately not corrected, a rise in the gold token's ranking indicates that the technique is moving closer to the correct answer.
50%UnverifiedMAPLE's beyond-sampling level detects whether KEE techniques can bring answers missed by sampling under a fixed sampling budget back within reach, thereby distinguishing between two entirely different root causes of failure: missing knowledge and sampling luck.
50%UnverifiedMAPLE's value lies not in proposing new KEE techniques, but in providing a set of analytical lenses for understanding and comparing existing techniques, helping researchers and engineers see the composition of the "net benefit" and make scenario-specific trade-offs.
50%UnverifiedAgainst the backdrop of increasingly homogenized large-model evaluation, decomposing evaluation dimensions itself may bring new discoveries, and MAPLE's research suggests that asking what happens behind the score is more meaningful than chasing leaderboard rankings.
50%