LLM judge
A paradigm in which a large language model is used as a judge to automatically score model outputs. It is typically used together with scoring rubrics.
Timeline (last 90 days)
LLM judges perform close to coin-flipping when detecting omission-type errors.
The LLM judge mechanism verifies the presence of information rather than its absence, so there is no clear anchor to trigger a check for omissions.
The solution to missed detections is to restructure the task rather than upgrade the judge model.
The research team used an LLM as a judge to identify the most and least strongly activated samples for each principal component, evaluate whether they suggest a transferable deception direction and assign scores, and select the highest-scoring subset of principal components to train the probe.
All Facts (4)
LLM judges perform close to coin-flipping when detecting omission-type errors.
50%UnverifiedThe LLM judge mechanism verifies the presence of information rather than its absence, so there is no clear anchor to trigger a check for omissions.
50%UnverifiedThe solution to missed detections is to restructure the task rather than upgrade the judge model.
50%UnverifiedThe research team used an LLM as a judge to identify the most and least strongly activated samples for each principal component, evaluate whether they suggest a transferable deception direction and assign scores, and select the highest-scoring subset of principal components to train the probe.
50%