Auditing LLM Annotation: Do Definitions, Label Names, or Demonstrations Drive Model Judgments?

Large language models are increasingly used to annotate text in the social sciences, but high accuracy does not mean the model is actually following the category definitions provided by researchers. A new paper uses a counterfactual audit method to reveal a neglected problem: model predictions may reflect prior associations with category labels or memorization of annotation demonstrations rather than an understanding of definitions.

The Attribution Puzzle Behind High Accuracy

The researchers' concern is straightforward: if a model has simply memorized the dataset's original label mappings, then even excellent test-set performance does not prove it can generalize definitions to new scenarios. To address this, the paper proposes a counterfactual audit—redistributing definitions among output labels to observe whether the model follows the new definitions or sticks to the dataset's original mappings.

This design cleverly separates two factors that are usually conflated: adherence to definitions versus reliance on label-example associations.

Counterfactual audit is a causal-inference-style evaluation method. Its core idea is to artificially intervene on a key variable (here, the correspondence between labels and definitions) and observe whether model outputs change with the intervention. In the LLM annotation setting, researchers shuffle and reassign the original label definitions—for example, attaching the definition of "economic issues" to the "social issues" label—and then test whether the model outputs according to the new definitions or follows the original label mappings from the training data. If the model sticks to the original mappings, its predictions mainly rely on memorized label-example co-occurrences rather than an understanding of definition semantics. This method effectively disentangles the two confounders of "definition adherence" and "label-association memorization," providing a deeper diagnostic signal for annotation quality than accuracy alone.

Models Rarely Follow Reassigned Definitions

On three political text datasets, experimental results show that models rarely follow reassigned definitions. More alarmingly, standard fine-tuning reinforces the original mappings without reliably improving adherence to definitions.

This means that the accuracy gains researchers see after fine-tuning may come from the model fitting the dataset's label distribution rather than understanding category definitions. In other words, accuracy metrics alone cannot distinguish "understanding the definition" from "memorizing the answer."

Retrieval-Augmented Annotation: Neighborhood Labels Are the Main Signal

The paper then investigates retrieval-augmented annotation. A counterintuitive finding is that retrieving the nearest training examples overall outperforms retrieving examples separately within each category, even though the former uses fewer demonstrations.

On two tasks with significant improvements, direct k-nearest-neighbor classification recovers 74% to 93% of the improvement, indicating that neighborhood labels provide most of the predictive signal. The gains from retrieval augmentation can largely be attributed to nearest-neighbor label information rather than deeper model understanding of definitions.

k-nearest neighbors (kNN) classification is a non-parametric classification method: for a text to be annotated, find the k closest examples in embedding space from the training set, and decide the prediction by voting among the labels of these neighbors. In the context of retrieval-augmented annotation, kNN classification can be viewed as a "pure memorization" baseline—it uses no category definitions provided by researchers and relies solely on the labels of nearest-neighbor examples. Thus, when the paper reports that "direct kNN classification recovers 74% to 93% of the improvement," it is actually quantifying how much of the retrieval-augmentation gain can be attributed to neighborhood label signals rather than model understanding of definitions. This comparison provides a key reference for judging whether retrieval augmentation truly improves definition adherence.

Relabeling Demonstrations Can Redirect Predictions

Experiments also find that relabeling fixed demonstrations can redirect model predictions; and when demonstrations are aligned with reassigned definitions, the model's adherence to the new mappings increases significantly.

This points to a practical conclusion: definitions, category labels, demonstrations, retrieval, and training data need to be jointly aligned and validated. Interventions that merely pursue accuracy gains may not improve adherence to the category definitions (codebook).

Demonstration relabeling means keeping the text of in-context examples unchanged while modifying only their corresponding labels, and observing whether model predictions change accordingly. This operation directly tests the model's sensitivity to "example-label" associations: if predictions flip after relabeling, the model relies heavily on label signals in demonstrations rather than inferring solely from text semantics or definitions. In the paper, when demonstrations are relabeled to be consistent with reassigned definitions, the model's adherence to the new mappings improves significantly, indicating that aligning demonstrations with definitions can partially correct the model's attachment to original label mappings. This offers practitioners an actionable intervention: by carefully constructing demonstrations consistent with definitions, guide the model to more faithfully apply the category system provided by researchers.

Implications for Social Science Annotation Practice

This audit work reminds researchers using LLMs for text annotation that accuracy should not be the only quality metric. When model predictions may be driven by semantic associations of label names or the proximity of demonstrations, researchers need additional audit tools to confirm whether the model is truly applying the definitions they provided.

The counterfactual definition-reassignment method offers an actionable testing approach: if the model still outputs the original labels after definitions are swapped, its "high accuracy" deserves a question mark.

One-line Summary

Counterfactual audits reveal that high LLM annotation accuracy may stem from label memorization rather than definition adherence.

Paragraph Summary

The paper uses a counterfactual audit method—redistributing definitions among output labels—to test whether large language models annotating social science text follow category definitions or rely on memorized label-example associations. On three political text datasets, models rarely follow reassigned definitions; standard fine-tuning reinforces original mappings without improving definition adherence. Gains from retrieval-augmented annotation mainly come from neighborhood label signals, with kNN classification recovering most of the improvement; relabeling demonstrations can redirect predictions, and adherence improves significantly when demonstrations align with reassigned definitions. The conclusion: accuracy gains do not equal improved definition adherence; definitions, labels, demonstrations, retrieval, and training data must be jointly aligned and validated.