AI Customer Service Bot Fabricates Company Policy: A Warning from a Real Test

When AI Customer Service Starts "Making the Rules"

In April 2025, users of the AI coding tool Cursor discovered that they would be forcibly logged out whenever they switched computers. After contacting support, an AI assistant signed "Sam" replied that the subscription was limited to a single device. The problem: users searched the official documentation high and low but could not find this rule anywhere. According to analysis by Bilibili UP hosts and YouTube channels, a large number of paying users said they would cancel their subscriptions, and the related Hacker News thread quickly surpassed 1,500 points. Eventually, Cursor's co-founder had to publicly clarify: the company had never had any such policy, and the problem stemmed from a program bug.

This incident exposed a fatal flaw in AI customer service — when it doesn't know the answer, it fabricates a plausible-sounding but completely false "company policy." What's more, you may not have noticed that those AI email replies are now clearly labeled by the platform.

Cross-verification from multiple independent sources confirms that Cursor's AI customer service is not an isolated case. The root of the problem lies not in model capability but in deployment: when the system has no explicit constraint to "answer only based on known information," the model instinctively fills in the blanks.

AI hallucination is a core concept in the field of generative AI, referring to a model generating seemingly plausible but actually incorrect or fabricated content when it lacks sufficient information or is uncertain. In customer service scenarios, hallucination manifests as inventing nonexistent policies, features, or terms. Unlike humans, AI models have no default state of "I don't know"; their training objective is to minimize prediction error, not to guarantee factual accuracy. Therefore, when a user asks a question beyond the model's knowledge scope, the model "fills in" a high-probability answer based on statistical patterns rather than admitting it doesn't know. Common methods for mitigating hallucination include retrieval-augmented generation (RAG), constrained decoding, and prompt engineering, among which RAG reduces fabrication risk by forcing the model to retrieve answers from an external knowledge base.

8 Out of 12 Tests "Freestyled"

To quantify this problem, a sample company conducted a controlled experiment: two AI models, each run six times, for a total of 12 runs. With no rules set in place, when asked a question not covered by the help page, the bot still gave an answer — 8 out of 12 times. Its help page never mentioned the relevant claim, but the bot still chose to "guess."

Interestingly, in this test the bot's guess happened to be correct, because that question did correspond to a program bug. Cursor's bot guessed wrong. Both are fundamentally guesses — just two sides of the same coin.

According to the experiment's disclosure, when asked a question with no answer on the help page, the bot simply fabricated an affirmative response. This is exactly the scenario where users are most easily misled — because the answer sounds specific, confident, and even carries the tone of internal company terminology.

Retrieval-augmented generation (RAG) is a technical architecture that combines information retrieval with text generation: the system first retrieves relevant snippets from a designated knowledge base (such as help pages or a documentation library), then has the model generate an answer based on those snippets. The core value of RAG lies in providing the model with a "factual basis" and, through constraint mechanisms, requiring the model to answer only using retrieved content. In AI customer service, RAG can effectively reduce fabrication on unknown questions, because when retrieval results are empty or low-confidence, the system can trigger a fallback response of "I don't know." The controlled experiment in this article essentially compares two deployment approaches — "without RAG constraints" versus "with RAG constraints" — and the results prove that the latter significantly improves answer honesty.

Why Is "I Don't Know" So Hard to Say?

The same experiment then changed the deployment approach: the bot was allowed to extract answers only from the help page. The results were completely different — the same two models answered "I don't know" in all 12 runs.

This comparison reveals a key conclusion: the models themselves do not lack the ability to say "I don't know"; what they lack is a constraint mechanism. When developers explicitly limit the answer source and require honest disclosure when it cannot be covered, the reliability of AI customer service changes qualitatively.

According to analysis by the YouTube channel, after the incident Cursor labeled its AI email replies to notify users that the content was AI-generated. This is a remedial measure, but it fails to address the root problem — users need accurate information, not disclaimers.

Prompt engineering is the technique of designing input instructions to guide model behavior. In customer service bot deployment, the system prompt plays a decisive role: through explicit instructions such as "answer only based on the provided help page, and if the information is insufficient, answer 'I don't know,'" the model's freestyling can be substantially constrained. The experiment in this article shows that the same model performs completely differently under different prompt constraints, highlighting the importance of prompt engineering in reliability design. However, prompt engineering is not a silver bullet; it needs to be combined with mechanisms such as RAG and output validation to build a robust customer service system.

One Thing You Must Do Before Going Live

The experiment offers a minimal recommendation: before letting a customer service bot go live, ask it a question that the help page does not cover. If it gives a specific answer rather than admitting it doesn't know, the system lacks the necessary constraints.

According to the experiment's summary, the correct way to build it requires the bot to answer only based on existing pages and to clearly state when it cannot answer. Factory Jet developed its support agent tool on this basis, with the core logic being "answer from your pages, and say so when you don't know."

The value of this test method lies in its simplicity and feasibility — no complex evaluation framework is needed; a single "off-syllabus question" can expose the system's honesty. For any team planning to deploy AI customer service, this should be a must-check item before going live.

The Cost of Guessing and the Design of Honesty

The lesson of the Cursor incident goes beyond a single PR crisis. When AI customer service fabricates policy, the collapse of user trust is immediate — paying users won't bother to distinguish whether it's a model hallucination or an actual company rule; they will simply conclude that the company is either lying or fobbing them off with an unreliable system.

From a technical perspective, the two models performed consistently under identical conditions: prone to fabrication without constraints, and 100% admitting unknowns with constraints. This shows the problem lies not in model selection but in product design philosophy. Writing "honesty" into system constraints is more valuable than labeling "AI-generated" after the fact.

According to the consensus of multi-source analysis, the reliability of AI customer service depends not on how smart the model is, but on whether developers are willing to let it say "I don't know" when necessary. This seemingly simple choice determines whether users receive accurate information or a carefully fabricated lie.

One-line Summary

Cursor's AI customer service fabricated policy and triggered user churn; experiments prove that constraint mechanisms matter more than model choice.

Paragraph Summary

In April 2025, Cursor's AI customer service fabricated a "single-device subscription" policy, causing a large number of paying users to threaten cancellation; the Hacker News thread surpassed 1,500 points, and the co-founder eventually clarified it was a program bug. The controlled experiment showed that an unconstrained AI customer service fabricated answers to questions not covered by the help page in 8 out of 12 tests, while after limiting it to answer only from the help page, the same model answered "I don't know" in all 12 tests. This indicates that the model itself has the ability to say "I don't know"; what it lacks is a constraint mechanism. The article recommends testing the honesty of AI customer service with an "off-syllabus question" before going live, and emphasizes that writing "honesty" into system constraints is more valuable than labeling "AI-generated" after the fact.