AI's Social Cognition Deficit: Why Models Consistently Over-Punish

From Telling Right from Wrong to Understanding Social Response Mechanisms

Over the past few years, AI alignment efforts have primarily focused on teaching models to recognize first-order social norms — that is, which behaviors are acceptable and which are not. Think clear-cut rules like "don't steal." AI Alignment refers to the research field dedicated to ensuring that AI systems' behaviors and goals remain consistent with human values and intentions. In recent years, techniques like RLHF (Reinforcement Learning from Human Feedback) have been widely used to train large language models to follow social norms. But these methods are fundamentally a process of encoding behavioral rules — highly effective when dealing with clear ethical boundaries, yet falling short when it comes to the vast gray areas and context-dependent judgments that pervade human society.

True social intelligence, however, goes far beyond this. Human social cognition isn't just about identifying whether a behavior violates a norm — it's about being able to predict who will enforce punishment and how they'll do it. When someone breaks a social rule, will bystanders publicly shame them, or choose to stay silent? Will the violator self-regulate out of guilt? These "norms about norms" are known as metanorms, and they determine how people respond when rules are broken.

The concept of metanorms was first introduced by political scientist Robert Axelrod in 1986. Through game-theoretic models, he demonstrated that first-order norms alone ("don't violate the rules") are insufficient to sustain social cooperation. Second-order norms — "punish those who fail to punish" — are also necessary. If someone witnesses a violation but does nothing, that inaction itself comes under social pressure. Axelrod's research showed that metanorm mechanisms are one of the key driving forces behind the self-maintenance of social order, a theory later widely cited in social psychology and evolutionary game theory.

A recent study on arXiv is the first to systematically evaluate large language models' performance on this kind of second-order social reasoning. In cognitive science, second-order social reasoning is closely related to "Theory of Mind" (ToM) — the ability to understand that others have independent beliefs, intentions, and emotional states, a capacity that typically matures in human children around ages 4–5. Second-order Theory of Mind goes even further: beyond understanding "what he's thinking," it involves inferring "what he thinks she will think." This recursive social cognition ability is the foundation for complex human social interactions such as negotiation, diplomacy, and understanding sarcasm. In recent years, multiple studies have begun evaluating LLMs' Theory of Mind capabilities, but most focus on belief reasoning rather than the norm enforcement prediction that this study addresses. The researchers proposed an innovative evaluation framework that reveals deep-seated deficits in how current AI understands social rule enforcement mechanisms.

NormReact Dataset: Evaluating AI's Metanorm Reasoning Along Two Dimensions

The evaluation framework constructed by the research team unfolds along two core dimensions:

Emotional Assessment Dimension

The first dimension concerns emotions — what emotional reactions do violators and bystanders have when a norm is violated? Human emotional responses to violations are not black and white; they are modulated by numerous contextual factors.

Behavioral Response Dimension

The second dimension concerns specific actions — what do people actually do in response to violations? The researchers designed two new classification tasks:

• Predicting violator self-regulation: Whether the violator will constrain their behavior due to guilt or shame
• Predicting bystander other-regulation: Whether observers choose to intervene, condemn, or remain silent

To support the evaluation, the team released a multi-perspective dataset called NormReact, containing 450 norm violation scenarios. These scenarios were manually annotated, covering emotional and behavioral responses across variables including violator gender and observer social closeness. NormReact's design reflects the "situationism" research tradition in social psychology — the idea that human behavior is determined not only by internal character but is also strongly influenced by situational factors. The "social closeness" variable introduced in the dataset (e.g., stranger, acquaintance, close friend) corresponds to the concentric circles of interpersonal relationships described by Dunbar's number theory: humans show significant differences in emotional investment and behavioral responses toward people at different social distances. While 450 scenarios may not be a massive scale, the carefully controlled variables give the dataset high internal experimental validity, enabling effective detection of systematic biases in models along specific dimensions.

Key Findings: AI Paints a Harsher Social Picture

The results carry significant cautionary implications. In testing six mainstream large language models, the researchers found a consistent trend: current LLMs portray a social world that is far harsher than reality.

Systematic Overestimation of Punitive Tendencies

The most prominent issue is that models tend to over-predict negative sanctions. In many scenarios where humans would typically choose "inaction" — tolerating, exercising restraint, or looking the other way — models predicted condemnation, shaming, or even harsher punishment.

The technical causes of this bias are likely multifaceted. First, during the RLHF training process, human annotators tend to reward models for giving "safer" and "more conservative" answers, and this preference gets internalized by the model as excessive vigilance toward norm violations. Second, there is a significant "negativity bias" in internet training data — on social media and in news coverage, public condemnation and punitive responses to violations are far more likely to be recorded and shared than silence and tolerance. This distributional skew causes the "social response baseline" learned by models to be inherently tilted toward severity. Furthermore, models lack embodied experience, and cannot perceive — as humans do through lived experience — the vast number of everyday social interactions where "nothing happens."

This means AI has a distorted understanding of how human social rules are enforced: it overestimates the prevalence of punishment while underestimating the tolerance, restraint, and relational trade-offs that pervade real life.

Accuracy Deteriorates as Social Distance Increases

Another key finding is that model-human judgment alignment worsens as social distance increases. In other words, the more distant the relationship between characters in a scenario, the more AI predictions diverge from actual human responses.

This phenomenon relates to a classic sociological theory — the "strong ties" and "weak ties" framework proposed by Mark Granovetter in 1973. In human society, norm enforcement within close relationships tends to be more flexible: family members might address violations through hints rather than open condemnation, and friends may choose tolerance to preserve long-term relationships. Between weak ties or strangers, norm enforcement tends to be more formal and direct. When making these judgments, humans automatically draw on vast stores of implicit knowledge about relationship costs, expectations of future interactions, and emotional bonds. This "relational calibration" ability is an extremely refined component of human social intelligence — one that current large language models have clearly not yet fully acquired.

Potential Risks in AI Application Domains

The implications of these findings extend well beyond academia. The researchers point out that in norm-sensitive application domains, this bias could pose substantive risks.

Hidden Dangers in Conflict Mediation and Policy Simulation

Imagine an AI system used for conflict mediation: if it systematically overestimates the likelihood that parties will resort to punitive responses, it could intensify opposition and amplify conflict. Similarly, in policy simulation scenarios, a model that over-predicts punishment and underestimates tolerance would produce a distorted social picture, potentially misleading decision-makers.

Real-world norm enforcement is full of gray areas: people forgive because the other person is an acquaintance, stay silent to preserve a relationship, and exercise restraint in many situations where condemnation would be justified. This nuanced social intelligence is precisely what current AI lacks.

From Recognizing Norms to Understanding Enforcement

This research suggests that AI alignment work needs to expand from mere first-order norm recognition toward deeper metanorm reasoning. A truly socially intelligent AI must not only know "what is wrong" but also understand "how people actually respond to wrongdoing" — including those moments when they choose not to respond at all.

The True Test of Social Intelligence

The title "Beyond Right and Wrong" itself pinpoints the core issue: social cognition goes far beyond binary judgments of "right and wrong." As AI increasingly enters sensitive scenarios involving human social rules, its ability to accurately understand the real landscape of social rule enforcement will directly impact the soundness and safety of its decisions.

This study and the release of the NormReact dataset provide an important starting point for evaluating and improving AI's second-order social reasoning capabilities. It reminds the entire industry: making AI "stricter" is not the same as making it "smarter" — true social intelligence lies precisely in the subtle balance of knowing when to punish and when to show mercy.

Key Takeaways