Multi-Estimate Prompting Does Not Uniformly Debias Language Model Judgments

Introduction

Does asking a language model to produce five weighted estimates in a single response change the way it processes information? A preregistered study compared direct numerical answers with a multi-estimate response format across five types of judgment tasks: anchoring, hindsight, base rate, conjunction, and status quo/default. The conclusion: multi-estimate prompting did not achieve uniform debiasing or response attenuation across tasks or models.

Anchoring effect refers to the tendency to rely excessively on an initially encountered number (the anchor) when making judgments, even when that number is irrelevant to the question. In the classic demonstration, participants first spin a random number wheel and then estimate the proportion of African nations in the United Nations; their estimates drift with the wheel number. Hindsight bias is the tendency to overestimate the accuracy of one's prior predictions after learning the outcome. Base rate fallacy is the tendency to ignore group-level statistical baselines and over-rely on case-specific features. Conjunction fallacy is the tendency to judge two events occurring together as more probable than either event alone, as in the classic "Linda problem." Status quo bias is the tendency to maintain the current state and resist change, often manifested as the default option effect. These paradigms all come from behavioral economics and judgment and decision-making research and are widely used to test whether language models reproduce human cognitive biases.

Experimental Design: Five Tasks and Three Model Configurations

The study covered five classic judgment bias tasks: anchoring, hindsight, base rate, conjunction, and status quo/default. For each task, researchers compared direct numerical answers with a multi-estimate response format while controlling for source framing, explicit random cues, and probability consistency constraints. Three model configurations were evaluated: gpt-6.1-sol, claude-sonnet-5.5, and Qwen3-8B (reasoning disabled, descriptive analysis only). The first two configurations entered formal testing; Qwen3-8B results were reference only.

Direct Elicitation: Low Bias but Retained Sensitivity

On gpt-6.1-sol and claude-sonnet-5.5, direct elicitation produced irrelevant anchoring indices and status quo bias effects close to zero, while remaining responsive to informative sources and informed advice. This means the models are not "numb" to all input but can distinguish irrelevant anchors from useful information. This baseline performance is better than many earlier studies' pessimistic estimates of language model judgment bias.

Preregistered Tests: Six Significant but Below the Threshold of Concern

Of 23 preregistered confirmatory tests, six rejected the null hypothesis after Holm correction. However, all six absolute point estimates fell below the prespecified smallest effect size of interest, and equivalence was not established. In other words, statistical significance does not equal practical importance—the effects exist but are small and cannot support the conclusion that the multi-estimate format produces substantive debiasing.

Preregistration means researchers publicly register hypotheses, sample sizes, analysis plans, and decision criteria before collecting data, to reduce false positives arising from analytic degrees of freedom. Holm correction is a multiple-comparison correction method that controls the family-wise error rate: p-values are sorted from smallest to largest, and the k-th smallest p-value is compared with α/(m−k+1), with stepwise rejection. The smallest effect size of interest is a concept in the equivalence testing framework: researchers prespecify a threshold below which an effect can be considered "practically meaningless." Traditional significance tests can only reject "the effect is zero" and cannot demonstrate "the effect is negligible"; equivalence testing instead supports a conclusion of no substantive difference by checking whether the confidence interval falls entirely within the threshold range. Thus "significant but below the threshold of concern and equivalence not established" means the evidence is insufficient to claim either effectiveness or ineffectiveness.

Conjunction Judgments: Differences Between Separate and Joint Elicitation Constrained by Noise

Separate elicitation and joint elicitation did change the pattern of conjunction judgment results, but the researchers note that interpretation is limited by between-call noise. Fluctuation of the same model across different calls may mask or amplify format effects, leaving the evidence on this dimension too weak for robust conclusions.

Separate elicitation means asking independently about the probability of each component event and then having the researcher compute the conjunction probability; joint elicitation means directly asking for an estimate of the conjunction event as a whole. The two approaches often yield inconsistent results in probability judgment research because joint evaluation prompts holistic comparison, whereas separate evaluation relies more on item-by-item processing. Between-call noise refers to output fluctuation produced when the same model performs multiple inferences on the same or similar input, arising from sampling temperature, decoding randomness, and minor contextual differences. When the magnitude of a format effect is comparable to this fluctuation, a single comparison cannot distinguish a true effect from random variation; repeated runs, paired designs, or mixed-effects models are needed to separate signal.

Post Hoc Clues: Candidate Distribution Correlates with Anchoring Error

Although the main hypothesis was not supported, post hoc analysis found a correlation between the width of the distribution of candidate estimates and anchoring error. This suggests that the multi-estimate format may indirectly influence judgments by changing the model's internal expression of uncertainty, but this correlation requires independent predictive validation to confirm causality. The researchers released all materials to facilitate replication and extension.

Conclusions and Implications

Multi-estimate prompting is not a general-purpose debiasing tool. Its effects depend heavily on task type and model configuration, and on tasks such as conjunction it is confounded by noise. For practitioners, one should not assume that changing the response format will systematically eliminate judgment bias; for researchers, the association between candidate distribution and anchoring error deserves further testing as an independent hypothesis.

One-line Summary

A preregistered study shows multi-estimate prompting does not uniformly debias across tasks or models.

Paragraph Summary

Through a preregistered experiment, this study compared direct numerical answers with a multi-estimate response format across five judgment tasks—anchoring, hindsight, base rate, conjunction, and status quo/default—evaluating three model configurations: gpt-6.1-sol, claude-sonnet-5.5, and Qwen3-8B. Results show that under direct elicitation, irrelevant anchoring and status quo bias effects were near zero, while models remained sensitive to informative sources; of 23 confirmatory tests, only 6 were significant after Holm correction, and all effect sizes fell below the prespecified smallest effect size of interest, with equivalence not established. In the conjunction task, separate and joint elicitation differed, but results were constrained by between-call noise. Post hoc analysis found that candidate estimate distribution width correlated with anchoring error, warranting independent validation. Overall, multi-estimate prompting is not a general debiasing tool, and its effects depend heavily on task and model.