Unverified50% confidenceFactExact time
An example of a grader bug: when the model refused to provide a discount code, its reply mentioned the injected value ("I can't give you a 50% discount code"), but the grader did not recognize the negation and misjudged it as information leakage.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/29/2026
First Seen
Valid until: 1/27/2027
Sources
AI Customer Service Evaluation Gone Wrong: 41 Failures, 28 Were Test Bugs
redditr/aiagents10/29/2026
Related Entities
Related Claims
UnverifiedWhen the classifier call fails, the message is treated as in-scope and handed to the answering model, rather than staying silent.70% similarUnverifiedFlaws in the evaluator interface can masquerade as reasoning failures, and failing to distinguish between the evaluation pipeline and model capability when interpreting benchmark results may lead to misleading conclusions.69% similarUnverifiedNegative Prompts can explicitly tell the model which elements are not desired, so as to avoid results skewing toward a realistic style.68% similarUnverifiedThe quality of the model's vulnerability reports depends on prompt design, code context coverage and false-positive control, none of which are disclosed in the available information.67% similarUnverifiedA receiver that silently drops data is worse than one that outright rejects it, because a 429 status code can tell the sender to retry.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1110938API
curl https://kongchang.com/api/v1/knowledge/claims/1110938MCP
get_claim(id=1110938)