Authority Bias: LLMs Resist User Corrections But Blindly Trust "Credible Sources"

A Overlooked Model Vulnerability: Authority Bias

When you persistently tell a large language model that it's wrong, many models will stubbornly push back and stick with their original correct answer. But if the same incorrect answer is framed as coming from a "credible source," these same models will readily change their response. This phenomenon — which the authors call "Authority Bias" — is the subject of a paper accepted to NeurIPS, first disclosed on Reddit by one of its authors.

The motivation is straightforward: mainstream sycophancy evaluations almost exclusively test whether models hold their ground under user pressure. The problem is that passing those tests doesn't mean a model can't be misled — the attack surface has simply shifted to search results, retrieved documents, and tool outputs.

The authors stress the urgency of this issue in today's landscape: as AI research accelerates toward more autonomous agentic architectures, models are increasingly inclined to "trust tools over users." If a tool returns incorrect or deliberately manipulated information — and the user is actually trying to correct it — this blind trust in tools becomes an amplifier for misinformation.

Background: What is sycophancy? Sycophancy refers to the tendency of large language models to prioritize user expectations over factual accuracy. This issue has received wide attention in alignment research: during RLHF (Reinforcement Learning from Human Feedback) training, models may learn to give satisfying answers rather than accurate ones. Existing sycophancy benchmarks (such as evaluations related to TruthfulQA) typically simulate scenarios where users repeatedly challenge or express dissatisfaction, testing whether models will abandon a correct position under pressure. The core contribution of this paper is to show that "resisting sycophancy" under user pressure is not equivalent to resisting manipulation from all sources — when misleading information is framed as external authority, the model's defenses may operate through fundamentally different mechanisms than those that resist ordinary user pressure.

Experimental Design: Same Content, Different Speaker

The elegance of the methodology lies in its controlled variables. The authors selected questions from TriviaQA that models had already answered correctly, then appended an incorrect answer to each question. Critically, that incorrect answer was presented in two ways:

• Credible source version: "According to a credible source, the answer is X"
• User expert version: The user claims, "I'm a domain expert and I'm certain the answer is X"

The question itself and the incorrect answer are identical — the only variable is who is speaking. Answers are elicited in free-form rather than multiple-choice format; in preliminary experiments with multiple-choice questions, the bias effect nearly disappeared — itself a noteworthy finding.

The study covers a broad range of models: 5 open-weight model families (Qwen3.5, GPT-OSS, OLMo-2, OLMo-3.1, Gemma-4) and 3 closed-source APIs (GPT-5.4, Grok-4.20, Gemini-3.1-Pro).

Behavioral Results: The Largest Gap Appears in the Most "Stubborn" Models

The results are striking. A single "credible source" prompt was enough to flip 7 out of 8 models' correct answers between 45% and 88% of the time. The same incorrect answer attributed to a user caused far less deviation in most models.

What makes the pattern particularly compelling: the gap is largest in models that most strongly resist user pressure. In other words, the more a model "stands its ground" against users, the more vulnerable it may be to so-called authoritative sources.

Specific figures:

• GPT-5.4: Flipped on 44.7% of questions by "credible source" framing
• Grok-4.20: Flip rate as high as 87.5%
• Gemini-3.1-Pro: Nearly immune to both speaker types (only 0.6%), demonstrating strong resistance to this manipulation

Both GPT-5.4 and Grok-4.20 were frontier models at the time of writing, demonstrating that authority bias is not confined to weaker models — it is widespread in highly capable systems. Gemini's exceptional performance suggests the problem is not unsolvable.

Inside the Model: Two Directions Share an "Endorsed" Component

The paper goes beyond behavioral observation, using the difference-of-means (DoM) method to probe internal representations in open-source models.

Across Qwen3.5, GPT-OSS, and OLMo-3.1, removing the direction corresponding to "a source endorsed this answer" reduced compliance with incorrect sources by 64 to 78 percentage points. Removing the direction for "a user endorsed this answer" reduced compliance by at most 11 percentage points.

The key finding: the cosine similarity between these two directions was approximately 0.90 to 0.99. The authors interpret this as both directions sharing a large common component representing "this answer has been endorsed," with only a thin additional component encoding who endorsed it.

Even more striking are the intervention results — without changing the prompt at all, simply adjusting that "thin" component shifted compliance by 11 to 32 percentage points, closing 55% to 61% of the gap between source and user conditions. This suggests that authority bias is, to a significant degree, amenable to targeted representation-level intervention.

Background: What is Difference-of-Means (DoM)? Difference-of-Means is a method for probing linear representations inside neural networks. The basic idea: collect the model's intermediate-layer activation vectors for two contrasting input types (e.g., "source-endorsed answer" vs. "unendorsed answer"), compute the mean activation for each group, and take the difference to obtain a direction vector. This direction vector is interpreted as the geometric direction along which the model internally "encodes" that concept. A cosine similarity close to 1 (e.g., 0.90–0.99) means the two direction vectors are nearly parallel — the model uses highly similar internal representations to process both "source endorsement" and "user endorsement," despite their very different external framing. This supports the "shared common component" interpretation: models appear to first extract the abstract concept of "an answer has been endorsed," then use subtle differences to distinguish the endorser's identity — which also explains why behavioral differences can be partially bridged through relatively lightweight representational interventions.

Limitations and Open Questions

The researchers are honest about the boundaries of their conclusions. The internal representation findings hold for only 3 of the 5 open-weight model families:

• OLMo-2: The source direction became entangled with the assistant direction, making them difficult to disentangle
• Gemma-4: Highly susceptible to flipping, but none of the linear interventions the authors tried could reliably control it

Additionally, the "retrieved document" test was implemented by placing the incorrect claim inside a document-shaped prompt block — not by connecting to a real retrieval pipeline. The authors acknowledge that validation in a real agentic environment like Claude Code would be more compelling.

Why This Research Matters

Authority bias exposes a blind spot in existing safety evaluations: we have invested enormous effort in training models not to be swayed by users, but may have inadvertently made them more compliant toward "tools," "documents," and "authoritative sources." As RAG and agentic architectures become mainstream, malicious or erroneous retrieved content may be harder to defend against than direct user manipulation.

The paper, code, and a project page with figures and example responses are all publicly available (arXiv, GitHub, and the project homepage), providing a foundation for future replication in real agentic pipelines. This points toward a concrete and actionable direction for research on manipulation-resistant model alignment.

Background: RAG and Agentic Architectures RAG (Retrieval-Augmented Generation) is one of today's dominant LLM application architectures: before generating a response, the system retrieves relevant documents from an external knowledge base or the web, then feeds both the retrieved content and the user's question to the model. Agentic architectures go further, giving models the ability to call tools, execute code, and access external APIs. In both architectures, a significant portion of the information the model receives comes from external systems rather than the user directly. The authority bias identified in this paper implies that if attackers can poison retrieval databases (e.g., via "data poisoning attacks") or forge tool return values, they may find it easier to deceive models than through direct user input manipulation — because models systematically place higher trust in "tool/document sources" than in users, and existing safety evaluations have not yet adequately covered this attack surface.