[KongchangAI]
· 2 min read· 1,196 words

Value-Policy Dissociation in LLM Agents: Aware of Consequences but Choosing the Wrong Action

Value-Policy Dissociation in LLM Agents: Aware of Consequences but Choosing the Wrong Action

Research finds LLM agents internally know which action is better but often don't choose it—value-policy dissociation.

This study uses mechanistic interpretability to examine the root causes of decision failures in LLM agents across Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B. In Grid and WebShop environments, candidate action values are linearly decodable from intermediate hidden states even when the policy chooses worse actions, suggesting "value-policy dissociation." Natural Value Transport interventions influence action preferences more consistently, and a closed-loop controller improves task outcomes. The work suggests distinguishing representation failure from utilization failure when evaluating agents.

Represented but Not Used: Value-Policy Dissociation in LLM Agents

LLM agents often make poor choices in tasks requiring multi-step decision-making, but it is hard to tell whether they fail to represent future consequences, or represent them but fail to use them. A new study systematically investigates this question using mechanistic interpretability methods across three models—Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B—and introduces the concept of "value-policy dissociation."

What Is Value-Policy Dissociation

The research team brings the reinforcement learning concept of action value into the analysis of LLM agents: in a partially observable Grid environment and a simulated shopping environment WebShop, they find that the relative value of candidate actions can be linearly decoded from the model's intermediate hidden states. Crucially, this value information remains present in the representation even in states where the policy prefers a worse action.

In other words, the model internally "knows" which action is better, yet its output policy selects the worse one. This disconnect between value representation and policy choice is what is called value-policy dissociation. It reveals a previously overlooked failure mode: an agent's poor decisions may not stem from missing information, but from information not being effectively utilized.

Action value is a core concept in reinforcement learning, usually denoted Q(s, a), representing the expected cumulative return from executing action a in state s and then continuing to act according to the current policy. It characterizes "how good this action is," not "which action the policy will choose." In the context of LLM agents, researchers approximate the model's internal encoding of "what to do next" at some intermediate layer as an implicit Q value: if the model truly understands the task, its hidden state should contain information distinguishing the quality of candidate actions. Linear decodability means that a simple linear probe (such as logistic regression or linear regression) can read out this value ranking from the hidden state, which is regarded as evidence that the information exists in the representation in a relatively explicit and accessible form. Value being decodable but not adopted by the policy is precisely the mechanistic basis of "dissociation."

How to Verify the Relationship Between Representation and Choice

Merely demonstrating that value is decodable is not enough; the research further asks: does this representation actually participate in action selection? The team compares two intervention approaches.

One is direct intervention along the value readout direction, which proves limited and inconsistent in effect. The other is "Natural Value Transport" (NVT), which learns a transformation from the natural variations in hidden states arising from differences in action value. Results show that NVT can more consistently influence action preferences, with more pronounced effects in later layers.

This contrast indicates that the encoding of value information in the model is not a simple linear readout channel, but rather closer to a distributed representation that must flow between specific layers to influence decisions.

Intervention experiments in mechanistic interpretability follow the logic of "readout—intervention—behavior observation": first locate the representation direction encoding a certain piece of information, then artificially perturb that direction to see whether the model's output changes accordingly. A typical directional intervention approach is to add or subtract a scalar along the value readout vector, directly amplifying or suppressing activation in that direction. This approach assumes that information is read by the policy along a single linear direction, but in actual models the same information may flow in a distributed manner across multiple layers; single-point perturbations are easily "corrected" or bypassed by subsequent layers, so their effects are often limited and unstable. What distinguishes Natural Value Transport (NVT) is that it does not forcibly set an intervention direction, but instead learns a transformation mapping from the natural variations in hidden states arising from differences in action value, making the intervention closer to the model's own computational trajectory. This also explains why NVT is more effective in later layers—the closer to the output, the closer the representation is to the decision variables actually adopted by the policy.

Task Improvement from Closed-Loop Control

The research also tests a closed-loop controller: treating the value readout as a sensor and NVT as an actuator, the two work together to regulate model behavior. Experiments observe improvements in task outcomes.

The value of this design lies in verifying the causal chain—if the value readout merely passively reflects information and is irrelevant to decision-making, then using it to drive interventions should not bring systematic improvement. The closed-loop experiment provides behavioral-level evidence that "representations can be utilized," and offers a new engineering approach for agent alignment and controllability: rather than retraining the policy, read and guide internal value signals at inference time.

Closed-loop control borrows the "sensor—actuator" framework from cybernetics: the sensor measures system state (here, the value readout), and the actuator applies regulation (here, NVT intervention), forming a feedback loop rather than open-loop one-time intervention. In agent experiments, this means reading the value signal in the current hidden state at each step and adjusting model preferences in real time accordingly, rather than performing a one-time perturbation at some fixed layer. The methodological significance of the closed-loop design lies in establishing causal evidence: if the value readout is irrelevant to decision-making, using it to drive interventions would not bring systematic behavioral improvement; conversely, stable task improvement indicates that this representation is indeed on the causal chain of decision-making. This approach also echoes the broader alignment direction of inference-time intervention, i.e., guiding internal signals without modifying model weights.

Implications for Agent Research

The significance of this work goes beyond naming a phenomenon. It suggests that evaluating LLM agents should not only look at whether the final action is correct or wrong, but should also distinguish between two types of problems—"representation failure" and "utilization failure"—because their remediation paths are entirely different.

At the same time, the difference in effectiveness between NVT and readout-direction intervention shows that intervention methods in mechanistic interpretability need to be closer to the model's natural computational path; simplistic directional interventions may underestimate the usability of internal information. For building more reliable agent systems, understanding how value representations flow and at which layers they are adopted by the policy may be more critical than simply scaling up model size.

One-line Summary

Research finds that LLM agents internally know which action is better but often do not choose accordingly—this is value-policy dissociation.

Paragraph Summary

Using mechanistic interpretability methods, this study investigates the root causes of decision failures in LLM agents across three models: Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B. In the Grid environment and WebShop, the authors find that the relative value of candidate actions can be linearly decoded from intermediate hidden states; even when the policy chooses a worse action, the value information remains present, leading them to propose the concept of "value-policy dissociation." The research further compares intervention approaches and finds that direct intervention along the value readout direction has limited effect, while Natural Value Transport (NVT), learned from natural variations in hidden states, can more consistently influence action preferences, especially in later layers. A closed-loop controller using the value readout as a sensor and NVT as an actuator can improve task outcomes. The research suggests that evaluating agents requires distinguishing between representation failure and utilization failure.

#LLM agents#value-policy dissociation#mechanistic interpretability#Natural Value Transport#action value#agent alignment#inference-time intervention
Share:

Related articles