Unverified50% confidenceFactExact time
The LLM judge itself may make mistakes, including misjudging correct behavior as a violation, missing real violations, and giving inconsistent judgments for similar situations; the paper validates the judge using manually verified synthetic golden conversations, but this is costly and hard to scale.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedSome reviewers directly use LLMs to generate review comments, which tend to be superficially fluent but contain factual errors, making them particularly harmful for work involving mathematical proofs.79% similarUnverifiedLLM judges should not be the only evaluation method, because they are costly, have randomness and may be fooled by the Agent's output style.76% similarUnverifiedThe LLM-as-a-Judge method introduces additional inference costs and may suffer from an attacker-defender homogeneity problem—attackers can use linguistic unreadability techniques to deceive the model acting as judge.73% similarUnverifiedThe choice of LLM judge has a significant impact on the final research conclusions, and using a lower-quality LLM as the judge may overestimate the intensity of context-memory conflicts.73% similarVerifiedLLMs may make incorrect judgments due to hallucinations, prompt injection, or contextual misunderstanding, and without independent verification mechanisms, faulty reasoning can directly translate into erroneous production operations73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1074918API
curl https://kongchang.com/api/v1/knowledge/claims/1074918MCP
get_claim(id=1074918)