Persuaded, Not Informed
A research paper studying how language model agents in CRM scenarios are misled by interest-biased testimony. It reveals a systematic failure mode in which models favor biased statements over objective records when processing customer-related information.
Timeline (last 90 days)
The study "Persuaded, Not Informed" finds that language model agents treat one-sided assertions from interested parties as evidence, thereby approving transactions that should have been rejected based on company records.
The study explicitly states that neither increased model scale nor enhanced explicit reasoning capabilities provides resistance to this persuasion-based failure.
Models that only read the call transcripts passed 29 of the 31 contradictory cases.
This failure mode is highly consistent across seven models from four vendors, with the rate of being misled as high as 87% to 97%.
Among the 35 genuine failure cases, only 3 did not involve any assertions, indicating that the vast majority of failures stem from accepting biased statements rather than from missing information.
When models were provided with complete company records, strict accuracy actually dropped from 41% to 18%, while recall rose and precision collapsed.
When budget and timeline calculations were delegated to an external reliable process rather than inferred by the model itself, accuracy improved by as much as 42 percentage points on weaker models and only 2 to 5 percentage points on stronger models.
The prerequisite for this failure mode is that the policy must be precisely specified in the input, and it only holds when the rules are clearly visible yet the model still believes contradictory verbal assertions
The research team publicly released all evaluation artifacts for reproduction and verification
On the strongest models, the differences between experimental groups fall within confidence intervals, and researchers characterized the results as directional patterns and soundness properties rather than proven performance lower bounds
All Facts (10)
The study "Persuaded, Not Informed" finds that language model agents treat one-sided assertions from interested parties as evidence, thereby approving transactions that should have been rejected based on company records.
50%UnverifiedThe study explicitly states that neither increased model scale nor enhanced explicit reasoning capabilities provides resistance to this persuasion-based failure.
50%UnverifiedModels that only read the call transcripts passed 29 of the 31 contradictory cases.
50%UnverifiedThis failure mode is highly consistent across seven models from four vendors, with the rate of being misled as high as 87% to 97%.
50%UnverifiedAmong the 35 genuine failure cases, only 3 did not involve any assertions, indicating that the vast majority of failures stem from accepting biased statements rather than from missing information.
50%UnverifiedWhen models were provided with complete company records, strict accuracy actually dropped from 41% to 18%, while recall rose and precision collapsed.
50%UnverifiedWhen budget and timeline calculations were delegated to an external reliable process rather than inferred by the model itself, accuracy improved by as much as 42 percentage points on weaker models and only 2 to 5 percentage points on stronger models.
50%UnverifiedThe prerequisite for this failure mode is that the policy must be precisely specified in the input, and it only holds when the rules are clearly visible yet the model still believes contradictory verbal assertions
50%UnverifiedThe research team publicly released all evaluation artifacts for reproduction and verification
50%UnverifiedOn the strongest models, the differences between experimental groups fall within confidence intervals, and researchers characterized the results as directional patterns and soundness properties rather than proven performance lower bounds
50%