MAPLE: A Multi-Axis Parametric Knowledge Extraction Evaluation Framework Explained

What Does Aggregate Accuracy Mask?

Large language models absorb vast amounts of knowledge from training data, but their inherent probabilistic nature does not guarantee that this knowledge will surface in their outputs. To bridge this gap, Knowledge Extraction Enhancement (KEE) techniques—such as activation steering, decoding adjustment, and prompt engineering—modify the inference process, potentially reinforcing existing predictions or bringing elusive answers to the surface.

The problem is that current evaluations of these techniques mostly focus on aggregate accuracy. This single-dimensional scoring collapses "recovered failure cases" and "broken success cases" into one number, masking the diverse effects that these techniques produce. In other words, when accuracy goes up, you have no idea how many problems were actually solved versus how many new ones were created.

Parametric Knowledge refers to knowledge that a model learns during pre-training from massive text corpora and stores in its weights, as opposed to external knowledge obtained at inference time through retrieval or context. KEE techniques specifically target this type of knowledge: it exists in the model's parameters but cannot necessarily be reliably invoked.

Activation Steering is a technique that intervenes in a model's internal hidden-layer representations at inference time. Representative methods include Contrastive Activation Addition (CAA) and Inference-Time Intervention (ITI). These methods compare positive and negative sample pairs to identify direction vectors associated with target behaviors (such as truthfulness or safety), then add that vector back into the model's activation space along a specific direction, thereby shifting output tendencies without fine-tuning.

Decoding Adjustment differs from activation steering in that it does not touch model weights or internal representations, but instead modifies the token selection strategy during generation. The typical Contrastive Decoding approach runs a strong model and a weak model simultaneously, taking the difference in their log probabilities as the score for the next token, in order to suppress the perfunctory outputs commonly produced by strong models. DoLa (Decoding by Contrasting Layers) contrasts the output distributions of different layers within a single model, amplifying the factual knowledge carried by high-confidence layers while suppressing generalization errors that may arise from lower layers.

MAPLE's Core Design: Diagnostic Metrics Across Three Levels

To address the above blind spots, researchers proposed MAPLE—a unified framework for characterizing how KEE techniques affect parametric knowledge extraction. Its diagnostic metrics decompose effects across three levels:

Surface Level: Answer Recovery and Disruption

The most direct level measures answer recovery and disruption. Recovery refers to questions that were originally answered incorrectly becoming correct after KEE intervention; disruption refers to questions that were originally correct becoming incorrect. By counting these two directions separately, the "one-pot" problem of aggregate accuracy is avoided.

Probability Level: Gold Token Ranking Shifts in Unrecovered Failures

The second level delves into probability distributions, measuring whether the ranking of the correct token (gold token) has shifted in failure cases that were not recovered. Even if the answer is not ultimately corrected, a significant rise in the correct token's ranking indicates that the technique is indeed "approaching" the right answer—a latent improvement that is completely invisible in aggregate accuracy.

Gold Token in this context refers to the token corresponding to the ground-truth answer. At each generation step, the model computes a probability distribution over the entire vocabulary—the so-called logits distribution. The gold ranking is the position of the correct token in that distribution: a rank of 1 means the model places it first, while a lower rank indicates that although the model did not output it correctly, the internal probability mass has shifted toward the correct answer.

The value of this metric lies in providing a continuous signal, whereas recovery/disruption is a discrete binary outcome. A KEE technique may fail to turn any question from wrong to right, but if the gold ranking jumps from several hundredth place to the top ten across many failure cases, it means the technique is indeed pushing the model closer to the correct answer. Researchers can use this to determine whether the technique is merely being buried by the final sampling step—and whether a different decoding temperature or a larger sampling budget could convert it into actual recovery.

The beyond-sampling level further asks: under a fixed sampling budget (e.g., 3 or 5 responses), some failures are not because the model doesn't know the answer, but because the correct response simply wasn't sampled during limited stochastic sampling. At this level, MAPLE detects whether KEE techniques can pull these answers missed by sampling back into reach, thereby distinguishing between two fundamentally different failure sources: missing knowledge versus sampling luck.

Beyond-Sampling Level: Recovery of Answers Missed by Fixed-Budget Sampling

The third level focuses on whether answers missed by fixed-budget sampling can be recovered. This involves cases where the model fails to generate the correct answer within a limited number of samples, and whether KEE techniques can help these answers surface.

Cross-Evaluation of Six Techniques Across Three Model Families

The research team evaluated six KEE techniques across three model families, covering three major categories: decoding, activation steering, and prompting:

• Model families: OLMo2, Qwen2.5, Llama3
• Technique categories: decoding adjustment, activation steering, prompt engineering

This cross-model-family evaluation design exposes performance differences of different techniques across different architectures. The analysis reveals starkly different effects and trade-offs among techniques—some excel at recovering failure cases but have high disruption rates, while others are relatively mild but have limited recovery capability.

Why MAPLE Is Worth Paying Attention To

MAPLE's value lies not in proposing a new KEE technique, but in providing an analytical lens for understanding and comparing existing techniques. For researchers and engineers, this means:

• When choosing a KEE technique, you no longer look only at how much accuracy improved, but can see the composition of the "net gain"
• Trade-offs can be made for specific scenarios—if an application scenario has zero tolerance for errors, then techniques with high disruption rates are unsuitable
• Fine-grained metrics such as gold ranking shifts provide more precise optimization signals for technique improvement

From a methodological perspective, MAPLE reminds us that in today's increasingly homogenized LLM evaluation landscape, decomposing evaluation dimensions itself can lead to new discoveries. When everyone is chasing higher aggregate scores, asking "what happened behind this score" may be more meaningful than climbing the leaderboard.

One-line Summary

MAPLE decomposes KEE techniques' bidirectional effects on knowledge extraction across surface, probability, and beyond-sampling levels.

Paragraph Summary

MAPLE is a unified analytical framework for diagnosing the actual effects of Knowledge Extraction Enhancement (KEE) techniques, addressing the blind spot where existing evaluations over-rely on aggregate accuracy and mask the bidirectional effects of recovery and disruption. The framework decomposes effects across three levels: the surface level separately counts answer recovery and disruption, the probability level measures gold token ranking shifts in unrecovered failure cases, and the beyond-sampling level examines whether answers missed by fixed-budget sampling can be recovered. The study evaluates six KEE techniques covering decoding, activation steering, and prompting across three model families—OLMo2, Qwen2.5, and Llama3—revealing significant trade-offs between recovery capability and disruption rates. MAPLE does not propose new techniques, but rather provides an analytical lens to help researchers and engineers see the true composition behind accuracy changes and make more precise technique selection and improvement decisions.