PerSo
A benchmark for evaluating LLM social intelligence, constructed from real persuasive dialogues in the Reddit ChangeMyView community. It simultaneously evaluates two dimensions: cognitive intelligence, involving mental state inference, and behavioral intelligence, involving persuasion strategy selection.
Timeline (last 90 days)
PerSo is a new benchmark for evaluating the social intelligence of LLMs in real persuasion scenarios, with data sourced from the ChangeMyView (CMV) community on Reddit.
PerSo decomposes social intelligence into two dimensions—cognitive intelligence (the ability to infer others' mental states) and behavioral intelligence (the ability to choose appropriate social actions based on those inferences)—and tests both simultaneously in the same real-world scenario.
The PerSo research team built an LLM-in-the-loop evaluation pipeline to generate evaluation tasks and validated it through human annotation.
On behavioral tasks (choosing appropriate persuasion strategies), the strongest LLM slightly exceeded the best human performance observed in the study.
On cognitive tasks (inferring others' mental states), the best human participants still led the strongest LLM.
PerSo leverages the Δ mechanism of the ChangeMyView community, so that high-performing persuasion strategies are not the researchers' subjective judgments but have been validated through community practice; such conversation data with natural outcome labels is quite rare in the field of persuasion research.
All Facts (6)
PerSo decomposes social intelligence into two dimensions—cognitive intelligence (the ability to infer others' mental states) and behavioral intelligence (the ability to choose appropriate social actions based on those inferences)—and tests both simultaneously in the same real-world scenario.
50%UnverifiedThe PerSo research team built an LLM-in-the-loop evaluation pipeline to generate evaluation tasks and validated it through human annotation.
50%UnverifiedOn behavioral tasks (choosing appropriate persuasion strategies), the strongest LLM slightly exceeded the best human performance observed in the study.
50%UnverifiedOn cognitive tasks (inferring others' mental states), the best human participants still led the strongest LLM.
50%UnverifiedPerSo leverages the Δ mechanism of the ChangeMyView community, so that high-performing persuasion strategies are not the researchers' subjective judgments but have been validated through community practice; such conversation data with natural outcome labels is quite rare in the field of persuasion research.
50%UnverifiedPerSo is a new benchmark for evaluating the social intelligence of LLMs in real persuasion scenarios, with data sourced from the ChangeMyView (CMV) community on Reddit.
50%