Unverified50% confidenceFactExact time
When an LLM scores as a judge, even with temperature set to 0, the same conversation trajectory and judge model run five times still produces a different verdict once, and this non-determinism is unacceptable for a test suite that wants to be integrated into CI.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/15/2026
First Seen
Valid until: 1/13/2027
Sources
Related Entities
Related Claims
UnverifiedLLM judge results are affected by factors such as temperature, prompt wording, and model version updates, causing the same set of test cases to potentially produce different conclusions when run at different times.80% similarUnverifiedAmong the six safety questions in MT-Bench-PT, on two questions the LLM judge refused to give a score no matter what the evaluated answer was, silently reducing the effective sample size N under that category.75% similarUnverifiedThe LLM-judge approach may give inconsistent conclusions for the same suspicious text when the temperature parameter is non-zero, and attackers can wait for passage by repeatedly retrying.73% similarUnverifiedEvaluation solutions that rely on LLM judges face obstacles in CI/CD scenarios, because non-deterministic judgments cause pipelines to pass sometimes and fail other times.72% similarUnverified即便温度参数设为0，不同的系统提示词、对话上下文和模型版本更新都会导致LLM输出差异，使品牌可见度测量面临可复现性挑战72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1036222API
curl https://kongchang.com/api/v1/knowledge/claims/1036222MCP
get_claim(id=1036222)