Using LLM Disagreement to Focus Expert Effort: Cross-Model Divergence Accelerates Large-Scale Annotation Codebook Revision

The Core Bottleneck in Large-Scale Text Annotation

Applying expert knowledge to millions of documents is a common requirement in social science research, educational assessment, content moderation, and beyond. The key tool for this is the codebook — a rule manual that guides annotators (human or AI) on how to classify and code text. However, developing a robust, reliable codebook often takes months, making it the primary bottleneck in large-scale annotation projects.

A recent paper published on arXiv (arXiv:2609.26926) proposes an intriguing approach: leveraging "disagreements" among large language models (LLMs) to precisely identify where experts should focus their energy — compressing what would otherwise be months of codebook revision down to days, without sacrificing annotation quality.

Codebooks have a long history in qualitative research and content analysis. A complete codebook typically includes: definitions and boundary descriptions for annotation categories, judgment rules and decision trees, and canonical examples to calibrate annotator understanding. Traditional codebook development follows an "iterative reliability testing" process — annotators independently label the same batch of samples using an initial codebook, researchers calculate inter-rater agreement metrics (such as Cohen's Kappa or Krippendorff's Alpha), then revise the codebook based on disagreement cases, repeating until agreement reaches an acceptable threshold (typically Kappa ≥ 0.7). This process usually requires multiple cycles, each requiring experts to convene and discuss edge cases, making it extremely time-consuming. The emergence of LLMs has made zero-shot or few-shot automatic annotation possible, but output quality is highly dependent on the clarity of codebook instructions — making codebook quality an even more pressing issue in the era of AI-assisted annotation.

Using Cross-Model Disagreement to Pinpoint Key Problems

The core insight of this research is that when multiple LLMs strongly disagree on how to annotate the same document, it typically signals that the codebook rules for that area are ambiguous, unclear, or insufficiently covered. These "disagreement cases" are precisely the high-value areas where expert intervention is most needed.

In other words, rather than having experts read through and revise an entire codebook line by line, let LLMs run through the data using an early-version codebook first — automatically surfacing cases where models disagree — then focus expert attention on these genuinely challenging areas. This is a classic "strategic allocation of human effort" approach: ensuring that scarce, expensive expert time is spent where it matters most.

Comparing Three Expert Feedback Approaches

The researchers systematically compared three methods by which experts can provide feedback for LLM codebook revision:

Codebook Verifying

Experts edit and confirm LLM-generated codebook revision suggestions that are driven by cross-model disagreements. This approach positions experts as "reviewers," acting directly on the codebook text itself.

Question Answering

For each disagreement between LLMs, the system poses specific questions to experts, who then provide answers. These answers are subsequently used to guide codebook revisions. This approach breaks expert feedback down into focused question-and-answer units.

Rationale Labeling

Experts directly label disagreement cases and attach reasoning explanations for their annotations. This approach provides not just the correct answer, but also the underlying decision logic.

These three feedback methods differ fundamentally in cognitive load and information granularity. Codebook verifying requires experts to have a holistic understanding of the rule system, simultaneously evaluating the logical soundness and boundary coverage of LLM-generated revisions — the highest cognitive burden. Question answering decomposes complex judgments into discrete questions, lowering the cognitive threshold for individual decisions, but introduces the risk that question design quality heavily influences outcomes. Rationale labeling is rooted in the behavioral psychology methodology of "think-aloud" — requiring annotators to externalize their decision process while making judgments. This not only provides correct answers but also captures the reasoning paths experts use when facing edge cases, precisely the high-density contextual signals that LLMs need to update their internal "understanding."

Experimental Results: Rationale Labeling Performs Best

The study conducted experiments on thousands of tutoring-session transcripts, measuring LLM annotation accuracy against expert labels as a benchmark. The results are compelling:

• Rationale Labeling achieved the highest LLM annotation accuracy at 64.9%;
• Question Answering reached 60.5% in its best configuration;
• By comparison, the traditional expert-revised codebook approach achieved only 57.8%.

This means both rationale labeling and question answering outperformed the traditional approach of having experts directly revise the full codebook. Rationale labeling led the expert-revised codebook by approximately 7 percentage points.

This finding is somewhat counterintuitive: having experts directly rewrite the codebook seems like the most "direct" optimization path, yet it proves less effective than having experts provide labeled rationales for specific disagreement cases. The likely reason is that judgments and reasoning on concrete cases provide LLMs with richer, more contextually grounded signals, while revisions to abstract codebook rules often fail to cover all edge cases.

Viewing this result through a machine learning lens: when LLMs receive codebook instructions, they are essentially mapping abstract rules to concrete textual features. Abstract codebook revisions provide signals at the rule level, while rationale labeling provides a triple of "rule + instance + decision logic" — equivalent to embedding high-quality few-shot examples in the context window. Research indicates this form of supervision signal has a more direct activating effect on LLMs' instruction-following capabilities. Additionally, the 64.9% accuracy was achieved under the strict standard of comparison against human expert labels; for multi-category, high-ambiguity discourse analysis tasks, this figure is quite competitive, while also clearly marking the current capability boundary of automated annotation.

Implications for Large-Scale Annotation Practice

The value of this research lies not only in the specific numbers, but in revealing a new paradigm for human-AI collaboration: LLMs serve not just as annotation tools, but as "attention routers" that can proactively surface the most important issues for human experts.

For teams relying on large-scale text annotation, this offers several practical takeaways:

• Using disagreement among multiple models as an "uncertainty signal" is a low-cost, automatable quality diagnostic;
• When collecting expert feedback, annotating cases with rationales may be more effective than abstractly revising rules;
• Through this approach, codebook development cycles could potentially be compressed from months to days, dramatically reducing time costs.

Of course, this study is based on a specific corpus of tutoring session transcripts, and whether its conclusions translate directly to other text domains — news, legal, medical, and so on — requires further validation. The overall accuracy hovering just above 60% also indicates that automated annotation remains some distance from fully replacing human effort. But as a methodology for accelerating expert collaboration and optimizing human resource allocation, it provides solid empirical evidence for rethinking large-scale annotation workflows.