Reinforcement Learning Teaches LLMs Theory-of-Mind Reasoning: Rule Learning from Perception to Action

The Theory-of-Mind Dilemma: Models Know But Don't Act

Large language models often exhibit a puzzling phenomenon in Theory-of-Mind (ToM) reasoning tasks—they can correctly infer a character's mental state in a story, such as someone mistakenly believing the chocolate is still in the blue drawer, yet when asked where that person will go to look for the chocolate, the model gives an answer based on "objective facts" rather than the "false belief." In other words, the model knows the character's thoughts but doesn't use those thoughts to predict actions.

This reasoning gap is no accident. It reveals a fundamental problem: the model lacks an explicit and stable connection mechanism between perceiving information, forming beliefs, and driving actions. A new study attempts to bridge this gap using reinforcement learning.

Theory-of-Mind (ToM) is a core concept in cognitive science, referring to an individual's ability to understand that others hold beliefs, intentions, and emotions different from their own. The classic research paradigm is the false-belief task: a child watches a character place chocolate in a blue drawer and leave, then another character moves the chocolate to a green drawer while they're away, after which the child is asked "where will that character look for the chocolate." Children who recognize the false belief answer "the blue drawer" because they understand the character holds a belief that doesn't match reality. This ability typically emerges at ages 4-5 and is regarded as an important milestone in social cognitive development. In AI, ToM is used as a test to evaluate whether models truly understand mental states rather than merely performing pattern matching. The "know but don't act" phenomenon in large language models corresponds precisely to the long-debated "belief-action separation" problem in cognitive science: even if a system can represent mental states, an additional mechanism is still needed to connect representation with behavioral output.

Connecting Perception and Action with If-Then Rules

The core idea of this research is straightforward: if the model cannot spontaneously chain perception, belief, and action together, then explicitly teach it such rules. Researchers generate concise if-then rules, such as "if the character did not see the object being moved, then she will search for it in its original location," then use Group Relative Policy Optimization (GRPO) to train Qwen2.5 and Llama3 series models, evaluating performance on four ToM benchmarks.

GRPO plays a key role here. Traditional supervised fine-tuning merely has the model imitate labeled answers, while GRPO optimizes the policy by comparing the relative quality of multiple generated outputs for the same question, pushing the model to actively discover what kind of rule generation approach leads to better reasoning performance. This is closer to letting the model "learn methods" rather than simply "feeding it answers."

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm proposed by DeepSeek^Ad, specifically designed for alignment and reasoning optimization of language models. Unlike traditional PPO, which requires training an additional value network (critic), GRPO samples multiple outputs for the same prompt and uses the relative quality of outputs within the group as a baseline to estimate the advantage function, thereby eliminating the overhead of a value model. This "intra-group comparison" mechanism is especially well-suited for ToM tasks: given the same story scenario, the model can generate multiple candidate rules, and the algorithm reinforces the policy by comparing which rule leads to more accurate action predictions. Compared to supervised fine-tuning, which directly targets labeled answers, GRPO's reward signal acts on the entire generated sequence, encouraging the model to explore "how good rules should be written" rather than merely replicating the distribution of correct answers. This characteristic gives it an advantage on tasks requiring structured reasoning rather than factual recall.

Why Rule Learning Outperforms Chain-of-Thought

The experiment compared four strategies: direct prompting, supervised fine-tuning (SFT), chain-of-thought reasoning (think reasoning), and rule generation under GRPO training. Results showed that GRPO combined with rule generation achieved the best performance across all four ToM benchmarks.

This result is worth pondering. Chain-of-Thought (CoT) reasoning has long been regarded as a general-purpose method for enhancing complex reasoning in models, having them "speak out" intermediate steps. But in ToM scenarios, having a reasoning process alone is still insufficient—the model may "say" the correct belief yet "do" the wrong prediction, as the reasoning chain does not truly constrain the final judgment.

If-then rules, by contrast, provide a causal constraint structure. When the model is trained to generate and apply these rules, it is forced to establish a verifiable mapping between perceptual information and action prediction. Rules are checkable: given what the character saw and didn't see, the rule directly specifies what the action prediction should be. This transparency itself is an effective training signal.

Chain-of-Thought (CoT) reasoning was proposed by Google in 2022, with the core idea of having the model output intermediate reasoning steps before giving a final answer, improving performance on complex tasks through "step-by-step thinking." Its effectiveness has been widely validated on tasks including mathematics, logic, and multi-hop question answering—the principle being that decomposing problems reduces per-step reasoning difficulty while intermediate steps provide additional computational space for the model. However, CoT has an often-overlooked limitation: it provides no verifiable constraint on the reasoning process itself. The model may generate reasoning chains that appear plausible but are actually disconnected from the final answer—"saying one thing but doing another." In ToM tasks, the model can fluently narrate "the character believes the chocolate is in the blue drawer" while bypassing this belief in action prediction to appeal directly to objective facts. The methodological significance of the if-then rule approach lies precisely here: it constrains reasoning into testable condition-action mappings, where each rule is an independent verifiable unit, compensating for CoT's fundamental flaw of "unverifiable process."

Small Models Generate Rules, Large Models Benefit

The research also uncovered a noteworthy cross-model transfer phenomenon: rules generated by models with ≥3B parameters trained via GRPO can serve as transferable knowledge to improve GPT-4o-mini's downstream performance.

This means the rules themselves carry structured knowledge about ToM reasoning, and this knowledge is not confined to the model architecture that generated it. A rule set trained on Qwen2.5 or Llama3 can be directly utilized by larger or different model families. This has two practical implications: first, small models can become "rule factories," providing reasoning enhancement for stronger models at lower cost; second, certain aspects of ToM capability may be encodable as formalizable knowledge rather than only emerging through scale.

Of course, the specific magnitude and applicability boundaries of the transfer effect still require more experiments to define. The researchers did not disclose numerical details of the transfer gains in the abstract, which reminds us to maintain cautious expectations about real-world application benefits.

Transferable knowledge here refers to a structured rule set extracted from a small model's training process that can be directly utilized by another independent model to improve performance. This transfer typically takes two forms: injecting rules as contextual prompts into the target model (in-context transfer), or using rules as training data to fine-tune the target model (fine-tuning transfer). The cross-model transfer in this research is important because it suggests that certain aspects of ToM reasoning may have a "model-agnostic" structure—if rules were only effective within a specific architecture, it would indicate they capture model-specific representational biases; but if they can transfer across architectures, it indicates the rules capture the task's intrinsic logic. This overlaps with the idea of knowledge distillation but is not entirely the same: knowledge distillation transfers soft label distributions, while rule transfer conveys readable, editable symbolic constraints with higher interpretability and composability.

Implications for Theory-of-Mind Research

This work points to a broader question: should large language models' Theory-of-Mind reasoning capabilities "emerge" through continued scaling, or can they be "constructed" through explicit structured guidance?

The research results lean toward the latter—at least for this specific perception-belief-action reasoning chain, explicit rules are more effective than implicit scale growth. This aligns with recent discussions suggesting that "small models + good methods" may outperform "large models + weak methods" on specific reasoning tasks.

However, ToM encompasses far more than just the perception-action pathway. Whether more complex ToM scenarios—recursive nesting of beliefs, multi-layered deception, causal inference of emotions—can all be covered by if-then rules remains an open question. The scalability of the rule-based approach depends on whether researchers can design sufficiently rich and non-conflicting rule sets for different types of ToM reasoning.

For researchers and developers focused on AI reasoning capabilities, this study offers a clear operational approach: rather than expecting models to learn the complete chain from perception to action on their own, use reinforcement learning to help explicitly forge each link in that chain.