Position Bias
A systematic bias in which an LLM judge, in pairwise comparison, tends to select the candidate answer in a fixed position regardless of content quality; swapping the positions of the options can reverse the judgment.
Timeline (last 90 days)
In language model evaluation, position bias typically manifests as a preference for a fixed position (such as first or second), and this preference is unrelated to the text content.
A common method for identifying position bias is to swap the order of candidates and observe whether the choice flips; if the flip rate is significantly above the random level, position sensitivity is indicated.
Position bias means the judge model's judgment of information in the context is affected by where that information appears: content near the end is often given higher weight, while key warnings that appear early are diluted in long conversations.
All Facts (3)
In language model evaluation, position bias typically manifests as a preference for a fixed position (such as first or second), and this preference is unrelated to the text content.
50%UnverifiedA common method for identifying position bias is to swap the order of candidates and observe whether the choice flips; if the flip rate is significantly above the random level, position sensitivity is indicated.
50%UnverifiedPosition bias means the judge model's judgment of information in the context is affected by where that information appears: content near the end is often given higher weight, while key warnings that appear early are diluted in long conversations.
50%