In Multi-Turn Conversations, Whose Values Prevail? New Research on Inference-Time Alignment

Human values profoundly shape our expectations of language model behavior. Mainstream alignment methods treat values as a global objective, but the appropriate value orientation is actually individual- and context-dependent. Training-free alignment techniques—Inference-Time Alignment—can steer models toward target values during inference, yet such methods have mostly been evaluated only in single-turn conversations and a single steering direction (e.g., toward safety). A new study moves the test scenario into multi-turn conversations, pitting a user model and an assistant model against each other toward the same or opposite—but equally legitimate—values, exposing many issues that single-turn evaluations had masked.

What Single-Turn Testing Masked

The research team evaluated four categories of prompt-based and activation-based steering methods, covering three models and two sets of value trade-offs. Unlike prior work, they used a linear probe to continuously track alignment throughout the entire conversation, rather than only looking at the effect in the first one or two turns.

The key finding comes from the "opposing pull" scenario: when the assistant and user push in opposite directions, the side matching the assistant's baseline bias often ends up prevailing. In other words, the model's own default tendencies quietly "win" over the user's steering across multi-turn interactions.

More subtle is the temporal effect. Some steering that appears successful early in the conversation gradually decays or even reverses after a few turns. This means that judging whether an alignment method is effective based solely on single-turn or early-turn evaluation results can lead to overly optimistic conclusions.

Linear probe is a lightweight tool commonly used in interpretability research: it reads—without modification—the internal activation vector at a certain layer of the model (usually a middle or later layer), then trains a simple linear classifier on top of it to determine whether that vector carries a specific semantic property (such as "does the current response lean toward safety" or "does it favor a particular value"). Its advantage is extremely low training cost and virtually no impact on the generation process, so it can be read in real time at every turn of the conversation, like a continuously measuring thermometer attached to the model. It is precisely with its help that this study scored alignment turn by turn, revealing that alignment does not rise monotonically but may decay or even flip after several turns—a kind of "temporal curve" that is invisible when only looking at the final output or the first few turns. Likewise, the distinction between activation-based steering and prompt-based steering is worth noting: the former directly intervenes in the model's internal activation direction, while the latter influences indirectly through input text. The difference in their performance under multi-turn accumulated error is precisely the core of what this study compares.

Why "Reverse" Evaluation Matters So Much

Value alignment is rarely single-dimensional. The study set up paired value trade-offs, and both directions are considered equally legitimate demands—not a setting like "safe vs. unsafe" with a clear right and wrong, but rather a trade-off between two legitimate values.

Under this setting, which side the model favors is not inherently a matter of good versus evil, but a matter of preference. When the steering direction aligns with the model's baseline preference, the method appears effective; once the direction is reversed, the effect can drop sharply. This suggests to researchers and product teams:

• Alignment evaluation must cover both directions of a value trade-off, not just testing the "pull toward safety" direction;
• Evaluation must run through the entire conversation, not just the first few turns;
• The model's own baseline preference is a variable that needs to be explicitly measured and accounted for.

Baseline bias refers to the default inclination a model naturally exhibits toward a certain value dimension in the absence of any steering or any special prompt. It comes from the statistical distribution of pretraining corpora, the selection tendencies of instruction-tuning data, and the collective consensus of human annotators during preference alignment (such as RLHF), and thus is often an "invisible prior." In multi-turn conversations, this prior continuously reinforces itself through the model's ongoing reinterpretation of context: each turn's generation provides context for the next, and phrasing that leans to one side is repeatedly cited and amplified, eventually forming a kind of inertia. When the user's steering direction aligns with the baseline preference, the two stack, making it appear that "the steering worked"; when the direction is reversed, the user's external force must fight this inertia every turn, and once the steering signal is insufficiently strong or decays, the baseline preference regains dominance. This also explains why "value trade-offs" cannot be simplified into "safe vs. unsafe": the latter has a clear right and wrong, while the former is a trade-off between two equally legitimate demands, where which side the model favors is purely a matter of preference, not of capability or morality.

Implications for Inference-Time Alignment Practice

Inference-Time Alignment has long been viewed as an attractive approach because it requires no retraining and is flexible to deploy. But this study shows that it is not robust under the pressure of multi-turn, bidirectional real conversations.

For developers, the direct action recommendation is: incorporate multi-turn conversations and bidirectional value trade-offs into the routine evaluation process for alignment methods; for product teams, it is necessary to recognize that when user and assistant values conflict, the model's default tendencies may be the actual dominant factor.

The study itself does not offer a simple conclusion about "which steering method is best," but instead focuses on evaluation methodology—only when we test with conversations sufficiently long and sufficiently bidirectional can we see the true boundaries of inference-time alignment.

Inference-Time Alignment is an umbrella term for a class of techniques that require no retraining of model parameters and only apply intervention at the generation stage. Its appeal lies in the fact that the base model remains unchanged, and through prompt engineering, activation steering, decoding constraints, and other means, the target values can be flexibly switched at deployment time, avoiding the high cost of separately fine-tuning for each value orientation. But its cost is also obvious—the intervention is "externally superimposed," and the model's own parameterized tendencies remain unchanged, so when the intervention signal is diluted in multi-turn interactions, drowned out by context, or inconsistent with the model's internal representation direction, the effect is diminished. It is often compared with training-time alignment (such as RLHF and DPO), which writes value preferences directly into the weights, offering greater stability but less flexibility and making it difficult to dynamically adjust for individual users. The significance of this study lies precisely in delineating the capability boundaries of inference-time alignment: it is suitable for lightweight, short-range steering that is aligned with or neutral to the baseline; in scenarios of long conversations and bidirectional value conflict, inference-time intervention alone is not enough to ensure that user values ultimately prevail.

One-line Summary

In multi-turn bidirectional conversations, the model's baseline preference often overrides the user's inference-time value steering.

Paragraph Summary

This study extends the evaluation of Inference-Time Alignment from single-turn, unidirectional settings to multi-turn, bidirectional scenarios: a user model and an assistant model push toward the same or opposite but equally legitimate values, testing four categories of prompt- and activation-based steering methods across three models and two sets of value trade-offs, while using a linear probe to track alignment turn by turn. The results show that when the two sides push in opposite directions, the side matching the assistant's baseline preference often ultimately prevails; some steering that appears successful early on decays or even reverses after a few turns. This indicates that evaluation conclusions from single-turn or early-turn tests may be overly optimistic, and also reveals that the model's own default tendencies are an underestimated force in multi-turn interactions. The study thus offers a methodological recommendation: alignment evaluation should cover both directions of value trade-offs and the full conversation length, and treat the model's baseline preference as a variable that needs to be explicitly measured and accounted for.