AI Debates Are Value-Rich but Differently Organized: A New Framework Quantifies Value Expression and Organization

Introduction: The Blind Spot in AI Debate Evaluation

Current evaluations of LLM-generated debates mostly focus on argument quality—such as logic, persuasiveness, or factual accuracy—but rarely examine the value structure behind the generated arguments. A recent paper, Value-Rich, Differently Organized: Measuring Value Expression and Organization in AI-Generated Debates, points out that AI-generated debates may be no less value-expressive than human ones, but differ significantly in how values are organized. The study introduces a diagnostic measurement framework that, for the first time, systematically compares how values are expressed and organized in human reference debates versus AI-generated debates.

Framework Design: Separating Value Expression and Organization

Based on Schwartz's Basic-10 value theory, the study represents debates as multi-level value distributions: from argument units, utterances, and stances to the entire debate. The framework explicitly distinguishes two dimensions: value expression refers to the range of identifiable values in a debate; value organization refers to how these values are distributed and sustained across multiple rounds of dialogue. Accordingly, the researchers propose metrics such as value coverage, alignment, diffusion, and cross-round dynamics, providing operational tools for quantitative analysis.

Schwartz's Basic-10 value theory was proposed by psychologist Shalom H. Schwartz and is one of the most widely used frameworks in cross-cultural value research. The theory identifies ten basic values that are universal across cultures: self-direction, stimulation, hedonism, achievement, power, security, conformity, tradition, benevolence, and universalism. These ten values are arranged along two orthogonal dimensions: one axis contrasts "self-transcendence" with "self-enhancement," and the other contrasts "openness to change" with "conservation." Each value has a specific motivational goal—for example, "universalism" emphasizes understanding, tolerance, and protecting the welfare of all people and nature, while "power" focuses on social status and control over people and resources. The theory's strength lies in its cross-culturally validated empirical foundation and structured dimensions, enabling researchers to map arguments in debates onto comparable value categories and quantify their distribution. In this study, it is precisely this theory that allows the authors to encode debate discourse into multi-level value distributions and design metrics such as coverage and alignment.

Core Findings: AI Is Value-Rich but Organizationally Diffuse

Experiments across multiple generation systems show that AI-generated debates are generally value-expressive, with the number of activated value dimensions often equal to or exceeding that of matched human reference debates. However, AI exhibits a different pattern in value organization: its utterances are more locally value-diffuse, meaning a single turn may mix multiple values without focus. Moreover, source-blind judges can distinguish value clarity from value focus, indicating that human evaluators can perceive this organizational difference.

These findings challenge the traditional evaluation approach that focuses solely on whether values appear, emphasizing that value organization also affects debate quality. For example, a debate may cover multiple values, but if each argument jumps between different values, its persuasiveness and coherence may suffer.

Source-blind judges refer to judges who evaluate debate quality without knowing whether the debate text was generated by a human or an AI. This design aims to eliminate "source bias"—the possibility that judges may systematically bias their quality assessments based on prior knowledge of authorship (e.g., believing human debates are more coherent, or being harsher on AI-generated content). In the study, source-blind judges were asked to distinguish value clarity from value focus, and the results showed that they could perceive the diffuseness of AI debates' value organization without being influenced by source labels. This method draws on Turing-test-style evaluation logic but emphasizes fine-grained judgments on specific dimensions (here, value organization), thereby providing relatively clean evidence that "organizational differences are perceptible to humans."

Implications: Debate Evaluation Needs to Account for Value Organization

The study calls for debate evaluation to go beyond value presence and incorporate the dimension of value organization. For developers of AI debate systems, this means that while optimizing argument quality, they also need to attend to strategies for distributing and sustaining values across dialogue. The framework's open-source code and supplementary materials have been made public, providing a reproducible foundation for future research. Future work could explore how to guide AI toward more structured value organization to enhance the depth and consistency of debates.

Conclusion

This study reveals the richness of AI-generated debates at the value level and their differences in organization, offering new perspectives and tools for LLM debate evaluation. As AI applications in debates, negotiations, and similar scenarios deepen, understanding its mechanisms of value expression and organization will become increasingly important.

One-line Summary

Study reveals AI debates are no less value-expressive than humans but more diffusely organized, calling for evaluation to incorporate value organization.

Paragraph Summary

Based on Schwartz's Basic-10 value theory, the study proposes a diagnostic measurement framework that distinguishes value expression from value organization, and for the first time systematically compares value structures in human versus AI-generated debates. Experiments across multiple generation systems find that AI debates activate a number of value dimensions often equal to or exceeding human reference debates, but exhibit local diffuseness in value organization—single turns often mix multiple values without focus. Source-blind judges can perceive this organizational difference, indicating that value organization is a quality dimension independent of value presence. The study calls for debate evaluation to go beyond value presence and incorporate value organization metrics, and has open-sourced the framework code and materials.