LLM as Judge
A method that uses a large language model as a judge to evaluate AI outputs. It can understand semantics and assess dimensions such as accuracy, faithfulness and tone, but it is costly and non-deterministic, requiring human calibration.
Timeline (last 90 days)
LLM judge results are affected by factors such as temperature, prompt wording, and model version updates, causing the same set of test cases to potentially produce different conclusions when run at different times.
Each LLM evaluation requires an additional LLM API call, and costs grow linearly with the scale of testing.
Evaluation solutions that rely on LLM judges face obstacles in CI/CD scenarios, because non-deterministic judgments cause pipelines to pass sometimes and fail other times.
All Facts (3)
LLM judge results are affected by factors such as temperature, prompt wording, and model version updates, causing the same set of test cases to potentially produce different conclusions when run at different times.
50%UnverifiedEach LLM evaluation requires an additional LLM API call, and costs grow linearly with the scale of testing.
50%UnverifiedEvaluation solutions that rely on LLM judges face obstacles in CI/CD scenarios, because non-deterministic judgments cause pipelines to pass sometimes and fail other times.
50%