[KongchangAI]
ConceptAI judge / LLM-as-a-Judge

AI Judge

An evaluation paradigm in which a large language model scores or judges the output of another system according to predefined criteria. It is often used to scale up human review, but carries risks of correlation bias and selection bias.

Timeline (last 90 days)

Nov 16

In clinical AI iteration pipelines, using AI judges (models that score another system's outputs according to scoring criteria) to scale clinical review has become common practice.

Unverified50%
Nov 16

When teams use AI judge evaluations to select prompts or train models, the judge is no longer a neutral external observer but becomes part of the intervention.

Unverified50%
Nov 16

Any change made based on AI judge evaluation results needs to be accompanied by an independent clinical test.

Unverified50%
Nov 16

Correlated bias may exist between the judge and the model, i.e., the judge and the model share the same kind of blind spots or preferences, so the judge's high scores cannot independently prove the model's safety.

Unverified50%
Nov 16

When evaluation results are used for prompt selection or model training, the judge systematically misses certain types of errors; such errors are not penalized among the candidates, so the selected solutions are more likely to carry them.

Unverified50%

All Facts (5)

Source Articles