From DQI to C-DQI: Using LLMs to Auto-Generate Codebooks for Evaluating Parliamentary Discourse Quality

When Parliamentary Debates Meet Large Language Models: A New Paradigm for Discourse Quality Assessment

Assessing the discourse quality of parliamentary debates has traditionally relied on experts scoring manually according to theory-based codebooks. While rigorous, this approach has clear bottlenecks: rigid codebooks, high annotation costs, and unavoidable expert subjectivity bias. A study on an academic preprint platform proposes a complete protocol from DQI (Discourse Quality Index) to C-DQI (Computational Discourse Quality Index), attempting to use large language models (LLMs) to solve two problems at once: "where do codebooks come from" and "are annotations reliable."

The Discourse Quality Index (DQI) is a theoretical framework widely used in political communication and deliberative democracy research for systematically evaluating the quality of parliamentary or public debates. It typically decomposes discourse quality into multiple dimensions, such as the justification of arguments (whether based on evidence and logic), respect (whether personal attacks are avoided), constructiveness (whether alternatives are proposed), and clarity of position. Researchers write codebooks based on these dimensions, and human annotators score debate texts item by item. The value of DQI lies in operationalizing the vague concept of "good debate" into measurable indicators, but its dimensions derive from a specific theoretical tradition and may not fully capture the unique discourse conventions in different countries' parliamentary cultures—which is precisely the motivation for subsequent computational adaptation.

Three Generation Paths: Top-Down, Bottom-Up, and Hybrid

Starting from the widely used DQI framework, the study compares three strategies for LLMs to generate codebooks:

• Top-down (theory-driven): Let the model generate evaluation items directly based on DQI's theoretical dimensions;
• Bottom-up (data-driven): Induce a context-adapted evaluation system from 100 Israeli parliamentary records;
• Hybrid approach: Combine theoretical framework with data characteristics for generation.

Behind this design is a key insight: theory-driven codebooks may miss important dimensions in specific parliamentary contexts, while purely data-driven approaches may deviate from widely accepted evaluation standards. Results show that inductive, data-driven coding schemes achieve higher inter-model agreement (IMA) while maintaining validity, suggesting that letting models "grow" evaluation criteria from real corpora may be more aligned with practical needs than directly applying theoretical templates.

Inter-Model Agreement (IMA) draws on ideas from ensemble learning and multi-annotator agreement research. Its core assumption is that different LLMs carry different systematic biases due to differences in training data, architecture, and fine-tuning strategies. If multiple models annotate the same corpus according to the same codebook and produce highly consistent results, it indicates that the codebook's items are sufficiently clear and operational, not easily distorted by any single model's bias. IMA is typically measured by the proportion of agreement or statistical measures (such as Krippendorff's alpha) between pairs of models. It provides a proxy signal for quickly estimating codebook reliability without human annotation, but high agreement does not automatically equal high validity—multiple models may share the same bias.

How Do You Know If an Auto-Generated Codebook Is Good?

The reliability of a generated codebook cannot be judged solely by what the models themselves say. The study designed two evaluation paths:

External proxy indicators: Define a quantifiable external criterion and validate it through human annotation, ensuring that automatic evaluation aligns with human judgment.

Inter-model agreement (IMA): Leverage bias differences among different LLMs by having multiple models annotate the same corpus, using their degree of agreement to signal overall reliability. This idea is quite clever—if multiple models with different "biases" all reach similar conclusions, then the codebook's reliability is more trustworthy.

According to the paper, experiments on 100 Israeli parliamentary records show that the data-driven approach outperforms the theory-driven approach on IMA without sacrificing validity. This means the LLM-based codebook generation route has potential for large-scale political discourse analysis.

External proxy indicators in measurement theory refer to substitute variables that are related to the target construct but easier to observe or more objective. In this study, researchers needed to verify whether the auto-generated codebook actually measures "discourse quality" rather than some pattern the models invent on their own. The approach is to select an external variable theoretically related to discourse quality (such as some observable outcome of the debate or overall expert ratings), test whether scores derived from the auto-generated codebook correlate with this external variable in the expected direction, and cross-validate with human annotation. This method cannot fully replace rigorous human validity testing, but it provides a necessary validity safeguard in large-scale automatic evaluation, preventing codebooks from being highly consistent across models while deviating from the true construct.

Why This Matters

Assessing parliamentary discourse quality matters beyond academia. Effective democratic deliberation is foundational to citizen participation and the healthy functioning of legislative institutions, and scalable, low-bias assessment tools can enable researchers, media, and the public to examine the quality of political debates more systematically.

The role of LLMs here is not simply an "automatic scorer" but a participant in codebook design and a component of the evaluation protocol. This two-layer approach—"using models to generate standards, then using inter-model disagreement to test those standards"—provides a reusable framework for measurement problems in computational social science. Of course, inter-model agreement is only a proxy signal for reliability and cannot replace human validity verification; the validity of data-driven approaches also needs testing on more corpora and in more contexts.

For readers interested in AI applications in social science, the value of this work lies in advancing "automation" to the generation of measurement tools themselves, not just the annotation stage. When codebooks can adaptively generate alongside corpora, large-scale, cross-context comparative research on political discourse truly becomes feasible.