Helpdesk Bot Knowledge Base Pitfalls: Reusing Ticket Replies Directly Leaks Customer Privacy

Helpdesk Bot Knowledge Base Pitfalls: Reusing Ticket Replies Directly Leaks Customer Privacy

Support bot knowledge bases are often built by having agents save high-quality replies. After a ticket is resolved, the agent clicks save, and that answer enters the bot's retrieval pool. This seems efficient, but hides a serious problem.

Pierre-Laurent Medori, engineering lead at GoodBarber, shared a full audit: due to a filtering bug, 65 saved answers had gone unused by the bot for months, and were about to become active again after the fix. He read through each one and found widespread privacy and accuracy issues.

The "Toxicity" of Ticket Replies

The root cause: when saving, the form is pre-filled with the ticket itself — the customer's last message and the agent's reply. Unless the agent rewrites it, what gets saved is a reply addressed to one specific customer. Such replies contain:

• Identity information: customer name, email, app name, domain, greetings, and signatures.
• Account links: links pointing to a specific customer's backend, which the bot might send to other customers.
• One-off actions: "The issue is resolved," "I tested it," etc., which the bot cannot perform.
• Statements on behalf of the customer: such as affidavits drafted in the customer's name.
• Time references: like "last Sunday," already-expired deadlines.
• Assumptions: assuming the customer has a particular setup, without any conditions.

In addition, problems like copied titles, answers covering only part of the question, contradictory answers to the same question, and broken help links were also widespread.

Pre-filled form: When an agent clicks "Save to knowledge base," the system automatically fills the edit box with the current ticket's customer last message and the agent's reply. This design is meant to reduce repetitive typing, but it makes agents tend to submit directly rather than rewrite into a generic answer. Pre-filled content naturally carries the ticket's context, including customer identity, specific accounts, and one-off dialogue. To break this habit, either switch to a blank generic template, or force the agent to answer before saving: "Would this answer still hold if the bot sent it to another customer?"

Rewriting into Generic Answers

The author handled the most dangerous answers one by one. For example, the affidavit was rewritten as a checklist of proofs the store expects; "The issue is resolved" was reduced to generic advice: "Resave and check the preview"; a diagnosis for a specific site kept the technical explanation but removed site information. Within a day, about 30 were rewritten and 1 was deleted.

Affidavit: In some e-commerce or service scenarios, customers need to submit a written statement proving a certain fact (e.g., not receiving goods, authorization changes, etc.). Agents sometimes draft an affidavit on the customer's behalf and save it to the knowledge base, but such text is often written in the customer's first person and contains the customer's name, order number, and specific dates. If the bot reuses it verbatim, it will send another customer a legal document written under a stranger's name — meaningless and potentially a compliance risk. The author therefore rewrote it as a "checklist of proofs the store expects," keeping only generic guidance on what materials are needed and how to submit them.

Prevention Rules

To ensure future saved answers are clean, the team established 10 rules, the core ones being:

1. Saved answers must be generic: put yourself in the other's shoes and ask, if the bot copies this to another customer, is it still correct?
2. No customer data: names, emails, app names, domains, IDs, greetings, signatures.
3. No account links; describe menu paths instead.
4. Do not describe actions that were performed.
5. Do not include dates.
6. When the answer depends on specific circumstances, write out both branches.
7. Questions must be clear and complete; answers must cover all key points.
8. One topic, one saved answer; keep only one among duplicates.

Clean Answers Make Loose Thresholds Less Costly

After cleanup, here's how the bot used saved answers: 42% of replies cited at least one saved answer, drawn from more than 20 different answers. Matching was often loose: at a distance below 0.40, 9 of 12 were relevant; between 0.40 and 0.50 (where about 80% of injections occurred), most were irrelevant, and some answers were even attached to messages like "Hi." But the model mostly ignored irrelevant injections — 10 ordinary questions about push notifications received saved answers for different features, and none of the replies mentioned it.

The author points out that if the answers had not been cleaned, the same threshold might have served the affidavit to customers asking about similar store rejection issues, and sent a stranger's backend link to customers asking about domains. The risk is not in the threshold, but in what enters the pool.

Vector distance and retrieval thresholds: When the bot retrieves saved answers, it typically encodes the question and answer separately into high-dimensional vectors, then computes the distance between them (e.g., cosine distance or Euclidean distance). A smaller distance means closer semantics. The 0.40 and 0.50 mentioned in the article are such distance thresholds: only below the threshold is something treated as a candidate answer injected into context. The looser the threshold (the larger the value), the more recall, but the higher the probability of mixing in irrelevant content. Tightening the threshold may miss answers that could have been reused. Thus, threshold selection is essentially a trade-off between recall and precision.

Injection: This refers to the retrieval system concatenating the matched saved answer as context before the user's question, and then handing it to the LLM to generate a reply. The model may adopt, partially adopt, or completely ignore the injected content. Even if the injection is irrelevant, the model can sometimes rely on its own judgment to ignore it, but this is not reliable — once the injected content contains sensitive information or false premises, the model may still repeat it back to the customer.

Tool Improvement Suggestions

• Do not pre-fill saved answers with the raw ticket; switch to a generic template, or display a test question next to the save button.
• Check before saving: automatically flag names, emails, domains, account links, dates, "I tested," "resolved," etc.
• Review the pool before expanding retrieval: bug fixes, looser thresholds, and new embedding models can all push answers to customers that were never read in that role.

If you also turn tickets into bot knowledge, what does your save button do to the ticket?

One-line Summary

If helpdesk agents' saved ticket replies are not cleaned up, they inject customer privacy and one-off content into the bot knowledge base; the risk lies in the content, not the retrieval threshold.

Paragraph Summary

Using GoodBarber's audit as an example, the article reveals a common pitfall in helpdesk bot knowledge bases: when saving answers, the form is pre-filled with the original ticket, causing answers to contain sensitive or useless information such as customer names, emails, account links, one-off actions, and expired dates. Of 65 saved answers, about half needed rewriting and one was deleted. After cleaning them one by one, the bot used saved answers with a 42% citation rate; even with a loose retrieval threshold, the model ignored most irrelevant injections. The core conclusion: the risk is not in the threshold, but in what enters the pool. The team accordingly established 10 prevention rules and recommends automated checks before saving, switching to generic templates, and reviewing the pool before expanding retrieval.