How Do Prompt Perturbations Affect Bias and Hallucinations in Large Language Models?

Large language models (LLMs) are rapidly infiltrating a wide range of decision-support scenarios — from enterprise analytics to personal assistants — with people increasingly relying on their judgments. But as model complexity grows, one unavoidable question emerges: just how reliable are these models? A study published on arXiv (arXiv:2609.35804v1) addresses exactly this, examining how slight perturbations to input prompts affect model behavior across two dimensions: bias and hallucination.

The Core Question: Does Perturbing a Prompt Make Models More or Less Reliable?

"Prompt perturbation" refers to making subtle edits to an original question — such as rephrasing, restructuring sentences, or swapping synonyms — while keeping the overall meaning intact. Intuitively, a robust model should produce consistent answers despite these changes. In practice, however, minor input variations can cause significant fluctuations in output quality.

The goal of this research is to evaluate LLM robustness against perturbed inputs in decision-making tasks. The researchers focus on two key dimensions: whether perturbations amplify or reduce inherent bias, and whether they make the model more prone to generating hallucinated content that contradicts facts.

Bias in the LLM context typically refers to systematic skewing of model outputs toward certain groups, positions, or conclusions — for example, asymmetric treatment of gender, race, or political affiliation — often stemming from imbalanced training data distributions. Hallucination refers to the model producing content that contradicts facts with high confidence, including fabricated citations, incorrect figures, or nonexistent events. Both are central topics in LLM reliability research, with a key distinction: bias represents systematic, directional deviation, while hallucination manifests more as random, factual errors. In decision-support scenarios, both can have serious consequences — bias leads to unfair judgments, while hallucinations lead to decisions based on false information.

A Counterintuitive Finding: Perturbations Can Sometimes Reduce Bias and Hallucinations

The study's most notable conclusion runs counter to assumptions made in prior research. Conventional wisdom held that prompt perturbations would undermine model stability and amplify unreliable outputs. But this work finds that, for certain models, perturbations can actually mitigate bias and hallucinations.

This suggests that perturbation is not simply a source of noise — its effect on model output depends on the internal mechanisms of the specific model. For certain architectures, a rephrased prompt may help the model escape the traps embedded in the original phrasing, leading to more balanced and factually grounded responses. This finding cautions against the assumption that "input change" automatically equals "quality degradation."

Mechanistically, this phenomenon can be understood as follows: certain phrasings in the original prompt activate "shortcuts" baked into the model during training, causing it to default to fixed response patterns that often carry biased or hallucinated content. When the wording is changed, the model can no longer match those shortcuts and is forced into more general reasoning, improving output quality as a result. This echoes findings in adversarial examples research — perturbations are not always harmful, and their effects are highly dependent on a model's internal representation structure. This also explains why different model architectures respond so differently to the same perturbation.

Model Comparison: Claude 3 More Stable, GPT-3.5 More Volatile

The study conducted cross-model evaluations across multiple datasets. Results showed that Claude 3 performed more effectively across most dataset-represented tasks, demonstrating stronger robustness and consistency.

In contrast, GPT-3.5 exhibited notably volatile performance: competitive with other models in some scenarios, but significantly lagging in others. This "hit-or-miss" characteristic is precisely what warrants the most caution when deploying decision-support tools — a model that performs reasonably on average but fails in critical contexts can introduce unpredictable risks.

This comparison also illustrates that model selection cannot rely solely on overall benchmark scores; it must account for stability performance within specific application contexts.

Implications for Real-World Deployment: Rigorous Testing Is Non-Negotiable

The value of this research extends beyond academia. When LLMs are used as real-world decision-support tools, their reliability directly impacts the quality of final decisions. The researchers emphasize that rigorous testing and validation are essential to ensuring the reliability and effectiveness of such AI assistants.

For developers and organizations, several practical takeaways are worth noting:

• Robustness testing should be part of the evaluation pipeline: Testing only standard prompts is far from sufficient — actively introducing perturbed variants is necessary to assess model stability.
• Model selection should be task-specific: No single model is optimal across all datasets; task-specific validation is essential.
• Bias and hallucinations require ongoing monitoring: These issues shift with input changes and cannot be solved once and for all.

Common approaches to robustness testing include: synonym substitution (replacing keywords with semantically similar terms), syntactic transformation (switching between active and passive voice, or between questions and statements), detail addition/removal (inserting or deleting irrelevant background information), and stylistic variation (switching between formal and colloquial registers). Dedicated prompt perturbation evaluation frameworks — such as PromptBench — have also emerged in the field, enabling systematic generation of perturbed variants and quantification of model stability scores. Incorporating such tests into pre-deployment evaluation workflows is increasingly becoming an industry norm for responsible AI deployment.

Conclusion

This research adds a valuable new perspective to the increasingly important field of LLM evaluation. It reveals the complex, non-linear relationship between prompt perturbations and model bias and hallucinations, while reaffirming the significant differences in robustness across models. For teams looking to integrate AI assistants into high-stakes decision-making scenarios, understanding and testing these nuances is an indispensable step toward building trustworthy AI systems.