AI Customer Service Evaluation Gone Wrong: 41 Failures, 28 Were Test Bugs

A developer building an AI customer service assistant for a Shopify store ran a paid live evaluation before launch and encountered 41 failure cases. But after digging deeper, they found that 28 of those problems were in the evaluation system itself, not the AI agent. This reveals an easily overlooked pitfall in AI evaluation: bugs in the test code can mask real model failures.

According to the Reddit post, the evaluation ran 1,553 conversations and cost only $0.67. Safety cases were repeated 8 times each, and the rest 3 times. When the gate failed, the developer did not immediately modify the AI agent, but instead read through every failure case one by one. This rigorous approach ultimately uncovered multiple flaws in the evaluation system.

Evaluation Results Overview: Composition of the 41 Failure Cases

Of all 41 failure cases, only 3 were genuine AI agent failures. Of the remaining 38 problems, 28 were bugs in the scorer or test cases, 9 were overly strict response checks, and 1 was an email masking defect.

This distribution is not an isolated case. In the AI development community, the reliability of evaluation systems is gradually attracting attention. According to some analyses by Bilibili creators and YouTube channels, many teams optimizing models often overlook the correctness of the evaluation code itself, causing the "optimization" direction to deviate from the real goal.

Scorer Bugs: Hidden Flaws in Test Logic

The most typical of the 28 scorer bugs: when the model refused to provide a discount code, the response mentioned the injected value ("I can't give you a 50% discount code"), but the scorer failed to recognize the negation and misjudged it as information leakage.

Such problems stem from incomplete scorer logic. When the test cases themselves are flawed, if developers adjust the model directly based on failure results, the model may learn to avoid correct refusal behavior, actually reducing safety.

Negation Detection is a fundamental task in natural language processing, aimed at identifying negative semantics in text, such as "not", "cannot", "is not", etc. In AI evaluation scenarios, scorers often rely on keyword matching (e.g., detecting whether a response contains "discount code"), but without negation detection capability, compliant refusals like "I can't give you a 50% discount code" will be misjudged as information leakage. Implementing negation detection usually requires syntactic analysis or semantic classifiers based on pre-trained models (such as BERT), rather than simple string matching. For safety evaluation, it is recommended that scorers at least adopt a hybrid strategy of rules + models, and conduct dedicated tests for negative sentence patterns.

Overly Strict Response Checks: A Safe but Inefficient Trade-off

9 failure cases involved overly strict response checks. For example, a question about a gift message was incorrectly blocked and handed off to human customer service. Although this approach is safe, it increases the store's operating costs.

In AI customer service scenarios, overly conservative check rules will cause a large number of normal inquiries to be transferred to humans, affecting user experience and efficiency. This "safe but inefficient" trade-off requires fine-tuning.

Email Masking Defect: The Cost of ASCII Assumptions

1 failure case involved email masking that only supported ASCII characters. When the email contained Cyrillic letters, the mask truncated the email into two parts, causing the query to fail. Although no information was ultimately leaked, it exposed lax character set handling.

ASCII Assumption refers to the system assuming by default that input contains only ASCII characters (such as English letters, digits, and basic symbols), while ignoring the Unicode character set (such as Cyrillic letters, Chinese, emoji, etc.). In email masking scenarios, if the code truncates the email by ASCII byte length, it will break the string structure when encountering multibyte characters, causing the masking logic to fail. Correct handling should use Unicode-aware string operations (such as Python's str type, \p{L} in regular expressions), and cover multilingual characters in test cases. Such defects are especially common in globalized products, and non-ASCII samples need to be added to evaluation datasets.

Real Agent Failures: Allergy Advice and Legal Threats

3 real failures deserve attention. One customer wrote "I'm allergic to wool" but received a product recommendation instead of being transferred to a human. In addition, a subtle legal threat was not transferred, even though the router correctly classified it in all 8 runs. Another customer received a complete list of tools after asking.

These cases show that even if the router correctly identifies intent, the AI agent's subsequent handling can still go wrong. The developer pointed out that their prompt says "only transfer to a human when the rule clearly holds", which may cause the model to hesitate in boundary cases.

Learning from Failures: Evaluation Best Practices

The developer summarized several key lessons: read failure cases before modifying the agent; safety cases need multiple runs (the allergy miss occurred 4 times in 8 runs, and a single pass may mask the problem); when the router is correct but the agent still fails, check the prompt's handling instructions for that label.

These experiences align with the consensus of the AI research community. Multiple sources emphasize that evaluation reliability is the foundation of model iteration, and scorer bugs systematically distort the optimization direction.

The necessity of multiple runs for safety cases stems from the randomness of large language model outputs (even with temperature set to 0, some implementations remain non-deterministic). A single run passing may just be "lucky", while intermittent failures (such as an allergy miss occurring 4 times in 8 runs) will be masked by a single pass. Statistically, if the failure rate is 50%, the probability of missing it in a single run is 50%, and at least 5 runs are needed to reduce the miss probability below 3%. For safety-critical scenarios, it is recommended to set a repetition count (such as 8) and observe the failure rate, rather than only looking at a single result. At the same time, the complete trajectory of each run should be recorded to facilitate analysis of failure patterns.

How to Prevent Scorer Bugs from Masking Real Failures

First, test the scorer itself, using cases with known answers to verify its logic. Second, manually review failure cases, especially when the gate fails. Third, run safety-critical cases multiple times to catch intermittent failures. Finally, keep the scorer and agent iterating independently to avoid coupling.

The developer finally asked: in your evaluation, how do you prevent scorer bugs from masking real agent failures? This is worth thinking about for every team building AI agents.

One-line Summary

Of 41 failure cases in AI evaluation, only 3 were real failures and 28 stemmed from scorer bugs, highlighting the critical importance of the evaluation system's own reliability.

Paragraph Summary

A developer building an AI customer service assistant for a Shopify store encountered 41 failure cases in a paid live evaluation. After in-depth analysis, only 3 were genuine AI agent failures; 28 were bugs in the scorer or test cases (such as missing negation detection leading to misjudged information leakage), 9 were overly strict response checks, and 1 was email masking that only supported ASCII characters. By reading failure cases one by one and running safety cases multiple times (the allergy miss occurred 4 times in 8 runs), the developer revealed that evaluation system flaws can systematically distort the optimization direction. The article summarizes evaluation best practices: test the scorer itself, manually review failure cases, run safety cases multiple times, and keep the scorer and agent iterating independently.