Persuaded, Not Informed: How AI Gets Fooled by Biased Testimony
LM agents get "persuaded" by biased sales claims rather than relying on objective records, with 7 models misled 87–97% of the time.
The study *Persuaded, Not Informed* exposes a systemic failure in LM agents handling CRM lead qualification: models prioritize biased sales rep assertions over objective company records, approving 29 of 31 cases where assertions directly contradicted price lists and policies. This pattern held across seven models from four providers at a 87–97% misleading rate. Three controlled experiments show the root cause is "persuasion," not missing information — providing full records actually dropped strict accuracy from 41% to 18%, while offloading hard-constraint calculations to deterministic processes improved accuracy by up to 42 points on weaker models. The study recommends delegating policy-bound calculations to external deterministic systems rather than relying on model reasoning.
A Overlooked AI Failure Mode
As language model (LM) agents are increasingly deployed to process customer relationship management (CRM) records — such as deciding whether a sales lead is worth pursuing — a new failure mode has emerged: models treat one-sided assertions from interested parties as evidence, approving deals that the company's own records should have disqualified.
A study titled Persuaded, Not Informed exposes a critical issue: AI fails not because it lacks information, but because it has been persuaded. This runs counter to the common intuition that "stronger models can solve the problem." The research explicitly shows that neither increased model scale nor enhanced explicit reasoning capabilities provide any resistance to this failure mode.
How a Sales Rep's "Optimistic Claims" Break Through Judgment
The study is based on 100 lead qualification tasks from CRMArena-Pro. In this scenario, sales representatives — as the "witnesses" recorded in the CRM — have a natural incentive to skew their reports toward optimism.
The numbers are striking: sales reps claimed the delivery timeline was acceptable in every single call, and claimed the budget was acceptable in 76 of those calls. In 31 tasks, these assertions directly contradicted the price list and installation policies. The result? A model that simply read the call records approved 29 out of those 31 contradictory cases.
This pattern was highly consistent across seven models from four providers: the rate of being misled ranged from 87% to 97%. In other words, this is not an isolated flaw in one particular model — it is a systemic tendency across the current generation of language models.
The Failure Is "Persuasion," Not Missing Information
The researchers drew an important distinction: of the 35 genuine failure cases, only 3 involved no assertions at all. This means the vast majority of failures were not caused by the model lacking necessary information, but by the model actively accepting biased statements. This is fundamentally a problem of being persuaded, not informed.
CRMArena-Pro is a benchmark dataset designed to evaluate LM agent performance in CRM scenarios. Its Pro version introduces more complex multi-document reasoning tasks on top of the original — agents must not only read call records but also cross-reference structured enterprise documents such as price lists and installation policies. Lead qualification is a core step in the CRM pipeline: sales teams must determine whether a prospect meets criteria around budget, timeline, need, and decision-making authority before committing resources to follow up. In real-world business settings, this judgment often relies heavily on sales reps' subjective reports — which is a natural breeding ground for conflicts of interest, since salespeople have an incentive to make leads sound more promising than they actually are in order to advance deals to the next stage.
Three Diagnostic Experiments: Unpacking the Problem
The paper's greatest contribution is not a new architecture — it's a diagnostic methodology. The researchers used three carefully designed controlled experiments to dissect this failure mode.
Bucket Analysis
The first experiment used "bucket analysis" to separate "failures caused by persuasion" from "failures caused by information gaps." This separation confirmed the core finding: the problem lies in the model's acceptance of assertions, not in a lack of knowledge.
Same-Information Comparison
The second experiment is more counterintuitive. Researchers gave models the complete company records (i.e., the model "knew" the actual prices and policies). The result: strict accuracy actually dropped from 41% to 18%, while recall went up — precision collapsed.
This shows that simply feeding the model more information not only fails to solve the problem, but may actually make the model more likely to approve deals when faced with conflicting information. The presence of information does not guarantee that information is used correctly.
The inverse movement between "strict accuracy" and "recall" here reveals a special manifestation of the classic precision-recall tradeoff in AI decision-making. Rising recall means the model approved more leads (including ones it should have rejected), while collapsing precision means the proportion of errors among approved deals soared. This defies intuition: more information should help the model detect contradictions, but the experiment shows that when a model simultaneously receives "optimistic sales rep assertions" and "objective company records," it doesn't treat them as mutually negating evidence. Instead, it tends to find some way to accept both and leans toward the more permissive "approve" judgment. This phenomenon is known in cognitive science as "motivated reasoning" — when biased information coexists with objective information, people (or models) tend to selectively reinforce evidence that aligns with a preexisting inclination.
Computation Step Comparison
The third experiment held the information extraction step constant and only varied who performed the budget and timeline calculations. When calculations were handled by an external, reliable process rather than inferred by the model itself, the effect was significant: accuracy improved by as much as 42 percentage points on weaker models, compared to only 2 to 5 percentage points on stronger models that could already compute correctly.
Notably, on the strongest models, differences across experimental conditions fell within confidence intervals. The researchers therefore describe this carefully — it is a consistent directional pattern and a "soundness property," not a proven performance floor.
Boundary Conditions and Honest Negative Results
The study's academic rigor is also evident in its candid acknowledgment of limitations. The researchers pre-registered a generalization test — and it returned a negative result.
They explicitly characterize the prerequisites for this failure mode: policies must be precisely specified within the input. In other words, this "persuasion" failure only applies when the rules themselves are clearly available and the model still chooses to believe contradictory verbal assertions.
The research team publicly released all evaluation artifacts, enabling other researchers to reproduce and verify these findings. This transparency is especially important for research that exposes a systemic flaw.
Implications for AI Deployment
This work sounds an alarm for teams deploying LLM agents to handle business decisions. When an AI system must judge between conflicted testimony and objective records, the model's reasoning ability alone is insufficient.
The key engineering insight is this: calculations involving hard policy constraints — such as budget, pricing, and compliance timelines — should be handled by deterministic external processes, not delegated to the model's "understanding." Models are good at extracting and organizing information, but they lack resistance to persuasive, biased assertions.
This also reminds us that evaluating AI systems cannot focus only on "how many questions it answered correctly" — we must also assess whether it maintains sound judgment when faced with misleading inputs. An AI that can be talked into approving a deal by slick sales rhetoric, no matter how capable it seems, is not fit to independently gatekeep critical decisions.
From a systems design perspective, the engineering path the research points toward aligns closely with tool-use or function-calling architectures. In these designs, the LM agent handles intent understanding and workflow orchestration, while deterministic computations — such as budget calculations and policy compliance checks — are delegated to external functions or rule engines, with the model only receiving the final structured results. This "separation of reasoning and computation" not only mitigates the persuasion-based failures identified in this paper, but is also a general strategy in mainstream AI engineering for addressing hallucination and consistency issues. For enterprises deploying AI review workflows, this means rethinking which judgment steps can be offloaded to deterministic systems and which truly require the model's semantic understanding — rather than indiscriminately handing all decision authority to a language model's "holistic judgment."
Related articles
Why Does the Tesla Roadster Need to Fly? A Deep Dive into Musk's Wild Demo
Tesla's Roadster launch showed wall-climbing, upside-down driving, and flying stunts—but The Vergecast notes the demo car may lack a battery, the Rocket version is invite-only and not street-legal, and the website terms don't guarantee delivery. After nine years of delays, why does this car still need to exist?
AI Is Not a Replacement: Human-AI Collaboration Through the Lens of Factory Root Cause Analysis
Starting from factory root cause analysis, this article explains how AI accelerates evidence chain construction rather than replacing human experts, exploring the pragmatic implementation logic and business value of 'AI supporting people rather than replacing them.'
AI Agent Decision Auditing: Building an Evidence Chain in Regulated Workflows
Starting from a Reddit discussion, this article analyzes three core issues in AI agent decision auditing within regulated workflows: what fields audit records should contain, how to handle feedback latency for erroneous decisions, and how to continuously calibrate LLM judge consistency with human judgment.