PsyAgentBench
A benchmark framework for evaluating cognitive biases in large language models. It re-runs classical psychology experiments on LLM agents and employs a multi-dimensional factorial cross-design to distinguish genuine model biases from surface-level effects caused by training data contamination.
Timeline (last 90 days)
The PsyAgentBench study proposes that a model producing response patterns associated with a psychological effect and a model genuinely possessing that bias are two distinct propositions.
The PsyAgentBench study made public as many as 41,904 trial data records.
The PsyAgentBench study covers up to three open-weight model families.
PsyAgentBench employs a multi-dimensional factorial crossed design—including named vs. blind testing, standard vs. counterfactual versions, and persona manipulation—to guard against training data contamination.
The anchoring effect exhibits a knowledge-dependent characteristic: the effect is nearly zero when facing well-documented facts and nearly complete when facing fabricated quantities.
Adding just a single persona description instruction is sufficient to eliminate, weaken, or reverse these cognitive effects.
The study formalizes three failure pathways when transplanting psychological paradigms to LLMs—persona dominance, group collapse, and safety selection—two of which are empirically documented.
The study argues that characterizing LLMs with a single scalar bias susceptibility score conceals structural differences, and that results should instead be reported using replication profile reports.
All Facts (8)
The PsyAgentBench study proposes that a model producing response patterns associated with a psychological effect and a model genuinely possessing that bias are two distinct propositions.
50%UnverifiedThe PsyAgentBench study made public as many as 41,904 trial data records.
50%UnverifiedThe PsyAgentBench study covers up to three open-weight model families.
50%UnverifiedPsyAgentBench employs a multi-dimensional factorial crossed design—including named vs. blind testing, standard vs. counterfactual versions, and persona manipulation—to guard against training data contamination.
50%UnverifiedThe anchoring effect exhibits a knowledge-dependent characteristic: the effect is nearly zero when facing well-documented facts and nearly complete when facing fabricated quantities.
50%UnverifiedAdding just a single persona description instruction is sufficient to eliminate, weaken, or reverse these cognitive effects.
50%UnverifiedThe study formalizes three failure pathways when transplanting psychological paradigms to LLMs—persona dominance, group collapse, and safety selection—two of which are empirically documented.
50%UnverifiedThe study argues that characterizing LLMs with a single scalar bias susceptibility score conceals structural differences, and that results should instead be reported using replication profile reports.
50%