[KongchangAI]
· 1 min read· 850 words

AmbiSteer: Teaching Vision-Language Models to Ask When Uncertain

AmbiSteer: Teaching Vision-Language Models to Ask When Uncertain

AmbiSteer schedules activation steering with a linear classifier to mitigate VLM overconfidence under referential ambiguity.

Vision-language models are often overconfident in referentially ambiguous scenarios, defaulting to a single interpretation rather than acknowledging uncertainty. The AmbiSteer study explores whether activation steering can serve as an inference-time mechanism to selectively elicit ambiguity-aware behaviors. Experiments find that activation steering alone has limited effect and is model-dependent, while prompting is usually more effective; combining prompting with steering increases clarification and enumeration but also misfires on unambiguous inputs. AmbiSteer therefore introduces a linear classifier to predict input ambiguity and condition the steering direction, raising the proportion handled appropriately from 59.0% to 77.0%. This work offers a lightweight, easy-to-deploy inference-time solution to overconfidence in uncertain multimodal scenarios, leaving open questions on classifier accuracy, steering transferability, and task generalization.

Vision-language models (VLMs) often appear overconfident when visual evidence is insufficient, giving definitive answers rather than acknowledging uncertainty. A study called AmbiSteer proposes a lightweight framework that combines activation steering with prompting strategies to selectively elicit ambiguity-aware behaviors, raising the proportion of inputs handled appropriately from an average of 59.0% to 77.0%.

AmbiSteer research overview

The Problem: Overconfidence Under Referential Ambiguity

In visual question answering (VQA), referential ambiguity is a typical scenario: multiple entities in the image could all be the referent of the question. For example, when asked "What is that red object?" and there are two red objects in the image, the model should recognize that ambiguity exists—either asking for clarification or enumerating the possible referents—rather than defaulting to one of them and forcing an answer.

Current VLMs generally struggle to make this distinction, often directly giving a single interpretation. This overconfidence is especially dangerous in uncertain multimodal scenarios because it masks the model's actual cognitive boundaries.

Method: Prompting First, Steering Second, Classifier as Scheduler

The core question the research team poses is: can activation steering serve as an inference-time mechanism to selectively elicit ambiguity-aware behaviors? AmbiSteer's design unfolds across three levels.

Steering Alone Has Limited Effect

Experiments on three VLMs show that activation steering alone has limited effect and is highly model-dependent. By comparison, prompting is usually a more effective approach. This suggests that purely internal state intervention is insufficient to reliably change model behavior.

Activation steering is a class of inference-time techniques that can intervene on model behavior without retraining. The basic idea is: during the model's forward pass, locate an internal activation direction associated with a certain behavior (typically obtained via the mean difference of hidden states between positive and negative samples), then add or subtract a scaling coefficient along this direction on the activation vector at that layer, thereby enhancing or suppressing the corresponding behavior. Since only inference-time activations are modified, activation steering is extremely low-cost and can be switched on and off instantly, but its effectiveness depends heavily on the chosen layer, the quality of the direction estimate, and the linearity of the model's internal representations—which also explains why it has limited effect when used alone and varies across models.

The Double-Edged Sword of Combining Prompting and Steering

When prompting and steering are combined, the model's behaviors of requesting clarification and enumerating referents increase significantly. But the side effects are equally obvious: these behaviors are also triggered on unambiguous inputs, causing the model to be "overly cautious" even on clear questions.

A Linear Classifier for Conditional Scheduling

The key innovation of AmbiSteer is introducing a linear classifier to predict whether an input is ambiguous and to condition the steering direction accordingly. Only when the classifier judges the input to be ambiguous is the corresponding steering applied. This scheduling mechanism raises the proportion of inputs handled appropriately from an average of 59.0% to 77.0%, achieving selective elicitation while remaining lightweight.

A linear classifier here refers to a lightweight probe that takes the model's internal hidden states (or pooled representations) as input and outputs an ambiguity probability through a single linear transformation plus Sigmoid/Softmax. Compared with training a full ambiguity detection model, a linear probe has very few parameters, requires little training data, and can directly reuse the VLM's existing representations, thus matching the overall lightweight positioning of "inference-time intervention." Its accuracy directly determines the quality of scheduling: missed detections leave ambiguous inputs unsteered, while false positives apply unnecessary steering to clear inputs, producing "over-cautiousness."

Significance: A Lightweight Method to Mitigate Multimodal Overconfidence

The value of this work lies in providing a lightweight approach to the problem of model overconfidence in uncertain multimodal scenarios. Unlike solutions that require retraining or large-scale fine-tuning, AmbiSteer schedules steering directions via a classifier at inference time, making it lower-cost and easier to deploy.

The study also leaves some open questions: how the classifier's ambiguity judgment accuracy affects overall performance, the transferability of steering directions across different models, and the method's generalization ability on more complex multimodal tasks all merit further exploration. But as a foundational work, AmbiSteer establishes the feasibility of the path "recognize ambiguity—condition the response."

One-line Summary

AmbiSteer uses a linear classifier to schedule activation steering, mitigating VLM overconfidence under referential ambiguity.

Paragraph Summary

Vision-language models are often overconfident in referentially ambiguous scenarios, defaulting to a single interpretation rather than acknowledging uncertainty. The AmbiSteer study explores whether activation steering can serve as an inference-time mechanism to selectively elicit ambiguity-aware behaviors. Experiments find that activation steering alone has limited effect and is model-dependent, while prompting strategies are usually more effective; combining prompting with steering significantly increases clarification and enumeration behaviors but also misfires on unambiguous inputs. To address this, AmbiSteer introduces a linear classifier to predict whether an input is ambiguous and conditions the steering direction accordingly, raising the proportion of inputs handled appropriately from an average of 59.0% to 77.0%. This work offers a lightweight, easy-to-deploy inference-time solution to overconfidence in uncertain multimodal scenarios, and leaves open questions about the impact of classifier accuracy, steering transferability, and task generalization.

#AmbiSteer#Vision-Language Models#Activation Steering#Referential Ambiguity#Visual Question Answering#Multimodal Overconfidence
Share:

Related articles