Prompt Lottery: The Sensitivity Crisis of LLM Judges to Synonymous Rewriting

Introduction: Hidden Concerns of LLM-as-a-Judge

LLM-as-a-Judge (LLMaJ) has become a common paradigm for evaluating large language model performance, but most methods implicitly assume a critical premise: a single judging prompt can serve as a stable evaluation tool. However, a study titled "Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing" challenges this assumption. The study reveals that even validated, semantically equivalent but superficially differently worded prompt variants (i.e., synonymous rewrites) can cause LLM judges to give inconsistent verdicts. The researchers named this phenomenon Prompt Lottery.

LLM-as-a-Judge (LLMaJ) refers to the evaluation paradigm in which a capable large language model (such as GPT-4, Claude, etc.) acts as a judge to score or make preference judgments on another model's output. Common forms include pairwise comparison (judging which of two responses is better), pointwise scoring, and reference-answer-based scoring. Compared with human evaluation, LLMaJ is low-cost and highly scalable, and has been widely used in model leaderboards (such as automated judging variants of Chatbot Arena), RLHF reward modeling, and academic benchmarks. However, its validity relies on several implicit assumptions: the judging model can understand the task intent, the scoring criteria are sufficiently expressed in the prompt, and the verdict is insensitive to the prompt's surface form. The Prompt Lottery study questions precisely the third assumption, pointing out that the stability of judging outputs may be overestimated.

Core Findings: How Synonymous Rewriting Shakes Evaluation Results

Experimental Design and Scale

The research team conducted a large-scale experiment covering diverse evaluation tasks, judge families, and model backbones. They compared original prompts with validated paraphrastic rephrasings, which were designed to keep the evaluation semantics unchanged and alter only the surface implementation.

Paraphrastic rephrasing in this context specifically refers to prompt variants that keep the evaluation semantics unchanged and alter only the syntactic structure or wording, such as switching between active and passive voice, synonym substitution, clause order adjustment, and adding or removing politeness modifiers. The study emphasizes that these rewrites were manually validated to ensure no change in scoring criteria was introduced, so any verdict differences can be attributed to surface form rather than task semantics. This differs from adversarial prompts or prompt injection: the latter deliberately induce erroneous behavior, whereas paraphrastic rephrasing simulates the wording differences naturally produced by different annotators and different template engines in real practice. Precisely because the rewrites are "harmless," the verdict drift they cause is more alarming—it shows that one source of LLMaJ variance is syntactic noise that cannot be eliminated through semantic validation.

Main Conclusions

• Evaluation result fluctuations: Synonymous rewriting can change LLMaJ evaluation results and, in some settings, even reverse the relative rankings between models. This means that model rankings based solely on a single prompt's judging results may be unreliable.
• Sensitivity far exceeding humans: A user study involving 10 human annotators showed that under synonymous rewriting, the consistency drop of LLMaJ judges was 6-9 times that of humans, and below the threshold generally accepted in usability research.
• Effectiveness of multi-prompt aggregation: The study shows that through multi-prompt aggregation, emphasizing judgments that remain consistent across prompt variants can significantly improve robustness.

Methodology: From Measurement to Mitigation

Measuring Prompt Lottery

The researchers first quantified the impact of synonymous rewriting on LLMaJ judgments. They did not modify prompts arbitrarily but used validated, semantics-preserving rewrites, thereby ensuring that differences came only from surface wording. This rigorous design makes the conclusions more convincing: even when prompts are semantically equivalent, LLMaJ judgments drift.

Human Baseline Comparison

Through the participation of 10 human annotators, the study established a human consistency baseline under synonymous rewriting. The results show that LLMaJ judges are far more sensitive than humans, suggesting that current evaluation pipelines based on a single prompt may overestimate the true differences between models.

Mitigation Strategy: Multi-Prompt Aggregation

The researchers propose that aggregating judgments from multiple prompt variants and giving higher weight to judgments that remain consistent across variants can effectively improve evaluation robustness. This strategy does not rely on deliberate prompt optimization but reduces variance through an ensemble approach.

Multi-prompt aggregation draws on the classic idea of ensemble learning to reduce variance: construct multiple semantically equivalent but differently worded prompts for the same evaluation task, obtain judging verdicts separately, and then merge them through voting, averaging, or consistency weighting. The core intuition of consistency weighting is that if a judgment appears stably across multiple prompt variants, it is more likely to reflect the model's true capability rather than a prompt artifact; conversely, judgments that appear only under specific wording should be downweighted. This strategy is similar to self-consistency sampling, but the difference is that the dimension of variation is the prompt rather than decoding randomness. The cost is a multiplicative increase in judging calls, so in practice one must trade off robustness gains against inference cost, for example by enabling multi-prompt aggregation only for model pairs with close rankings.

Implications for AI Evaluation Practice

Limitations of Single-Prompt Evaluation

The Prompt Lottery phenomenon shows that treating LLMaJ as a stable evaluation tool carries risks. In model leaderboards, A/B tests, or academic comparisons, if only one prompt is relied upon, rankings may flip due to slight changes in wording. This is a warning for teams that rely on LLMaJ for decision-making.

Practical Value of Multi-Prompt Aggregation

Multi-prompt aggregation provides a simple and effective mitigation. Practitioners can introduce multiple semantically equivalent prompt variants into the evaluation pipeline and adopt a consistency weighting strategy to obtain more reliable model rankings. Although this increases computational cost, it may be worthwhile compared with the risk brought by incorrect rankings.

Future Directions

The study calls on the community to re-examine LLMaJ evaluation protocols. Future work can explore more robust aggregation methods, techniques for automatically generating semantically equivalent prompt variants, and using human consistency as one of the metrics for evaluating LLMaJ quality.

Conclusion

The "Prompt Lottery" study reveals a neglected vulnerability in LLM-as-a-Judge: synonymous rewriting is enough to shake evaluation conclusions. Through multi-prompt aggregation, we can hope to build a more robust evaluation system. In today's rapid iteration of LLMs, ensuring the reliability of the evaluation tools themselves is as important as improving model capabilities.

One-line Summary

Synonymous prompt rewrites can cause LLM judging results to drift and even reverse rankings; multi-prompt aggregation can mitigate this.

Paragraph Summary

This study reveals a "Prompt Lottery" phenomenon in LLM-as-a-Judge: semantically equivalent but differently worded prompt variants can cause LLM judges to give inconsistent verdicts and, in some settings, even reverse the relative rankings between models. Through a large-scale experiment covering diverse tasks, judge families, and model backbones, combined with a baseline comparison involving 10 human annotators, the study finds that LLM judges are 6-9 times more sensitive to synonymous rewriting than humans, with consistency drops below thresholds generally accepted in usability research. The study further proposes a multi-prompt aggregation strategy that significantly improves evaluation robustness by consistency-weighting judgments across multiple prompt variants. The conclusion is that single-prompt evaluation is insufficient to support reliable model ranking, and evaluation protocols need to incorporate prompt-variant ensembles.