Prompt Lottery
Refers to the phenomenon in which LLM-as-a-Judge evaluation is highly sensitive to semantically equivalent but syntactically different prompt rephrasings, leading to verdict drift and inconsistency.
Timeline (last 90 days)
A study titled "Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing" reveals that semantically equivalent prompt variants with different wording cause LLM judges to give inconsistent verdicts, and the researchers named this phenomenon Prompt Lottery.
In model leaderboards, A/B tests, or academic comparisons, if only a single prompt is relied upon, rankings may flip due to minor wording changes.
All Facts (2)
A study titled "Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing" reveals that semantically equivalent prompt variants with different wording cause LLM judges to give inconsistent verdicts, and the researchers named this phenomenon Prompt Lottery.
50%UnverifiedIn model leaderboards, A/B tests, or academic comparisons, if only a single prompt is relied upon, rankings may flip due to minor wording changes.
50%