PerSo: A New Benchmark for LLM Social Intelligence in Real-World Persuasion Scenarios

Social intelligence has always been a key yardstick for measuring whether AI can truly integrate into human social interaction. Large language models (LLMs) have already achieved impressive results on various exams and reasoning tasks, but when they face real interpersonal persuasion scenarios, can they accurately infer the other party's mental state and choose appropriate persuasion strategies accordingly? A new study called PerSo is precisely designed to answer this question.

What Is Social Intelligence? The Dual Requirements of Cognition and Behavior

The study breaks down social intelligence into two interrelated aspects. Cognitive intelligence refers to the ability to infer others' mental states—understanding what they are thinking, how they feel, and why they hold a certain position. Behavioral intelligence is the ability to choose appropriate social actions based on these inferences to achieve social goals.

Persuasion scenarios naturally activate both abilities simultaneously. To persuade someone to change their view, you need not only to understand the emotions and logic behind their position, but also to say the right thing at the right time. Previous research typically evaluated these two dimensions separately and often relied on synthetic scenarios, creating a clear gap from real-world interpersonal interaction. PerSo's breakthrough lies in examining both dimensions simultaneously within the same real-world scenario.

Real Data from the ChangeMyView Community

PerSo's data comes from Reddit's well-known ChangeMyView (CMV) community. Users in this community post their views and sincerely invite others to persuade them from different perspectives. Community rules require responders to provide in-depth arguments rather than simple rebuttals. These naturally occurring persuasion dialogues provide high-quality material for evaluating LLM social intelligence.

The research team built an LLM-in-the-loop evaluation pipeline to generate evaluation tasks, which was validated through human annotation. This means the evaluation tasks can be generated at scale while also being quality-checked by human judgment.

LLM-in-the-loop is a semi-automated paradigm for building evaluation data: large language models handle the heavy lifting such as initial screening, rewriting, candidate option generation, and difficulty stratification, while human experts are responsible for final annotation, correction, and quality control. Compared to purely manual construction, it can scale benchmarks by several orders of magnitude; compared to purely automated generation, it avoids the distribution shifts and factual errors common in synthetic data through human review. In PerSo, this pipeline's role is to convert the large volume of unstructured, colloquial persuasion dialogues from the CMV community into uniformly formatted, automatically scorable cognitive and behavioral questions—for example, extracting the OP's true stance and emotional cues from the original post to construct mental inference questions, and extracting effective persuasion strategies from real replies as correct answers for behavioral questions. It is precisely because human judgment is embedded in the process that PerSo can serve as a reliable evaluation benchmark while maintaining its scale advantage.

Evaluation Results: Scaling Helps, but Performance Diverges Across Dimensions

The study tested different LLMs, and the results revealed several noteworthy patterns.

Performance Grows with Model Scale, but Not Uniformly

Overall, model performance improves with parameter scale. This trend exists in both cognitive and behavioral tasks, but the magnitude of improvement is not consistent across dimensions. This means that simply scaling up models does not guarantee simultaneous improvement across all aspects of social intelligence.

The Strongest LLM Slightly Exceeds the Best Human Performance on Behavioral Tasks

On behavioral tasks—selecting appropriate persuasion strategies—the strongest performing LLM slightly exceeded the best human performance observed in the study. This is a notable milestone, indicating that LLMs have reached a fairly high level in strategy selection.

Humans Still Lead on Cognitive Tasks

However, on cognitive tasks—inferring others' mental states—the best human participants still outperformed the strongest LLM. This gap suggests that the ability to understand others' inner worlds may still be a weak point in current AI systems.

The evaluation methods for cognitive intelligence and behavioral intelligence are fundamentally different, which partly explains why their performance diverges. Cognitive tasks typically require the model to output inferences about others' mental states—for example, judging whether the OP holds a certain position due to identity, past hurtful experiences, or lack of information—and the answers tend to have relatively clear "correct" references that can be directly compared with human annotations. Behavioral tasks require the model to choose among several candidate persuasion strategies, with a more structured answer space where the model only needs the ability to recognize and match persuasion tactics (such as appealing to empathy, pointing out logical flaws, or providing counterexamples) to score. In other words, behavioral tasks are closer to pattern recognition and strategy retrieval, while cognitive tasks rely more on deep modeling of implicit context, emotions, and intentions. This distinction explains why LLMs can surpass humans in the behavioral dimension but lag behind in the cognitive dimension—the former can be learned from massive text corpora, while the latter depends on theory-of-mind capabilities that are difficult to directly distill from language data.

What This Benchmark Means

PerSo's value lies not only in providing a new evaluation tool, but also in revealing an easily overlooked fact: social intelligence is not a single-dimensional capability, but a set of interrelated yet unevenly developed skills. LLMs can perform exceptionally well at the behavioral level, even surpassing humans, but still fall short at the cognitive inference level.

For AI application developers, this means that when building systems involving interpersonal interaction, one cannot judge social capability based solely on a model's overall performance. A model that excels at strategy selection may still make mistakes when understanding users' true intentions. PerSo provides a more granular evaluation framework to help researchers pinpoint specific weaknesses in models.

For researchers focused on AI alignment and human-AI collaboration, PerSo also raises a deeper question: when AI can select effective persuasion strategies but may not truly understand the other party's mental state, what are the ethical implications of this persuasion capability? The answer to this question may be more worth pondering than the benchmark scores themselves.

The design of the ChangeMyView community itself is also noteworthy. The community mandates that responders must "directly challenge the OP's view" rather than make personal attacks, and has a special mechanism: when the OP believes a reply has successfully changed their mind, they can respond with a Δ (delta) marker. This mechanism gives "whether persuasion was successful" a traceable public signal, allowing researchers to distinguish which persuasion strategies are truly effective in real interactions. PerSo leverages exactly this—high-performing strategies are not the researchers' subjective judgments but have been validated through community practice. This kind of dialogue data with natural outcome labels is quite rare in persuasion research and is an important reason why PerSo has greater ecological validity compared to previous benchmarks that relied on laboratory scenarios or synthetic dialogues.

One-line Summary

PerSo benchmark uses real Reddit persuasion dialogues to evaluate LLM social intelligence, finding that behavioral dimensions already surpass humans while cognitive dimensions still lag behind.

Paragraph Summary

PerSo is a new benchmark designed to evaluate large language models' social intelligence. It breaks social intelligence into cognitive intelligence (inferring others' mental states) and behavioral intelligence (choosing appropriate social actions), and for the first time examines both simultaneously in real persuasion scenarios. The research data comes from Reddit's ChangeMyView community, with tasks generated through an LLM-in-the-loop pipeline and validated by human annotation. Test results show that model performance overall grows with parameter scale, but improvement across the two dimensions is uneven; the strongest LLM slightly exceeds the best human performance on behavioral tasks, while humans still maintain the lead on cognitive tasks. This benchmark reveals that social intelligence is not a single capability but an unevenly developed set of skills, offering AI developers and alignment researchers a more granular evaluation perspective and ethical questions worth pondering.