LLM Judge

Batch effect refers to the way other cases the judge sees in the same batch indirectly affect the log-probability distribution of the current case through the attention mechanism.

A method for diagnosing LLM judge variance is to run the same batch of inputs five to ten times in a row and observe whether the score distributions overlap on both sides of the pass line.

For tone judgments on customer service agents, pointwise scoring is used, comparing a single answer against a threshold; when ranking is needed, switch to pairwise comparison and run both orders, because models tend to favor the answer they read first.

In adversarial input scenarios, both typed and generative judges read the content being scored, and the content may carry instructions targeting the judge, so written reasoning cannot reliably expose prompt injection.

Using heterogeneous evaluators from different vendors and different training pipelines can reduce the correlation between errors

Requiring evaluators to provide reasoning alongside their scores helps expose whether evaluators are making judgments based on genuine content quality rather than surface features