Building an AI Customer Service Agent from Scratch: A 92% Intent Recognition Case Study

Many companies' first instinct when automating customer service is to write rules: match whatever the user says against keywords. The result is usually that a month later, intent recognition accuracy plateaus around 65%, 6 out of 10 conversations go nowhere, and task completion sits at just 40%.

A Bilibili creator shared an AI customer service agent solution that has already been deployed at multiple companies, handling over 100,000 conversations per day, achieving 92% intent recognition accuracy and 78% conversation completion, raising user satisfaction from 3.2 to 4.6 and saving companies more than 60% in labor costs. This system isn't built on buzzwords—it's held up by four engineering-focused key modules. Here's a breakdown following this practical path.

Understanding: Layered Intent Recognition with BERT

Rule matching has a low ceiling because real user expressions are too diverse. This system uses a BERT model for intent recognition, first classifying user questions into three major categories—inquiry, complaint, and transaction—then breaking them down layer by layer into specific business scenarios.

First determine "what do you want to do," then match "which specific matter is it." The advantage of this layered breakdown is that the classification boundaries at each layer are clearer, and model training converges more easily. The improvement from 65% to 92% accuracy comes mainly from here—replacing keyword matching with deep learning so the bot understands semantics rather than memorizing word lists.

BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model proposed by Google in 2018. Its core innovation is the bidirectional Transformer encoder—when understanding a word, it simultaneously uses contextual information from both its left and right sides. Traditional word embeddings (like Word2Vec) assign a fixed representation to each word and cannot handle polysemy; BERT dynamically generates word representations based on sentence context, so it can better capture semantic differences like "I want to check my order" versus "I want to complain about an order issue." For intent recognition tasks, the usual approach is to attach a classifier on top of BERT's output layer and fine-tune it with labeled customer service conversation data to adapt the model to specific business scenarios. Compared to keyword matching, BERT can generalize to unseen expressions—this is precisely the technical foundation for the accuracy jumping from 65% to 92%.

Remembering: A Dialogue State Tracker Manages Context

The biggest fear in multi-turn dialogue is "amnesia." If the user gave an order number earlier and later says "that product," the conversation breaks down if the system can't connect the dots. This system has a dialogue state tracker that specifically records order numbers, phone numbers, and referents the user has mentioned, always retaining the context of the last ten turns.

This module may seem unremarkable, but it's the prerequisite for multi-turn conversations to proceed continuously. Without state tracking, even the strongest intent recognition can only handle single-turn Q&A.

Retrieving Accurately: A Hybrid Approach of Vector Search Plus Exact Matching

Knowledge base retrieval is the second checkpoint for customer service system accuracy. This system uses a hybrid approach of vector search plus exact matching: when a sentence comes in, semantic similarity is computed first, then results are re-ranked by urgency and knowledge freshness.

An easily overlooked detail is weight decay—the weight of old knowledge older than 30 days is automatically reduced, preventing the system from responding to users with outdated policies. Many customer service bots give irrelevant answers not because the model is inadequate, but because the knowledge base is "answering new questions with old answers."

Vector search refers to encoding text into high-dimensional dense vectors through an embedding model, then measuring semantic similarity by computing cosine similarity or inner product between vectors. Unlike keyword search, which relies on literal matching, vector search can recall content with different wording but the same meaning, such as "how do I get a refund" and "what is the refund process." Common vector index structures include approximate nearest neighbor (ANN) algorithms like FAISS and HNSW, which can complete semantic search over large-scale knowledge bases in milliseconds. In a hybrid approach, exact matching handles structured queries like order numbers and product models, while vector search handles natural language questions—the two complement each other and significantly reduce missed and false recalls. Re-ranking typically introduces business rules (such as urgency) and timeliness factors to re-score candidate results.

Knowing When to Back Off: A Weighted Scoring Mechanism for Handoff to Humans

The fourth point is the most easily overlooked: the system must know when to hand off to a human. The trigger conditions in this solution are low confidence, rising user emotion, and repeated failures on the same question—all three are jointly assessed through a weighted scoring model, and once the threshold is reached, it decisively hands off to a human.

The creator's exact words were "AI isn't meant to replace people, but to reserve people for the moments that need them most." This design philosophy is worth referencing for any team building a customer service agent—handing off to a human isn't a failure, it's part of the system design.

Confidence score in classification models usually refers to the model's probability output or normalized score for the predicted category. When the BERT intent classifier's highest category probability for the current user utterance falls below a certain threshold, it means the model is "uncertain" about the user's intent, and continuing to auto-reply carries higher risk. In engineering practice, the confidence threshold needs to be calibrated according to business tolerance: too high a threshold leads to frequent handoffs and increased labor costs; too low lets the bot force answers and harms user experience. In practice, AB testing and online feedback data are commonly used to dynamically adjust the threshold. Jointly modeling confidence with emotion recognition and repeated failure counts is essentially a multi-signal fallback strategy that ensures the system makes conservative decisions when uncertain.

Launch Is Just the Beginning: Engineering Is the Watershed

What enables the solution to run in production isn't just the model. The original article mentions microservice architecture, containerized deployment, real-time monitoring and alerting, and continuous iteration through AB testing. This engineering support determines whether the system is "a working demo" or can "handle 100,000 conversations a day."

For teams struggling with customer service costs, the reference value of this breakdown lies in this: each of the four modules has a clear engineering goal and can be implemented in phases, rather than rebuilding the entire system at once. The creator also mentioned that the next direction is multimodal affective computing and deep integration with large language models.

FAQ

Q: What scale of company is this solution suitable for? The original article doesn't give an explicit size threshold, but judging from the data of 100,000 conversations per day and 60% labor cost savings, it's suitable for teams whose customer service volume has already created cost pressure.

Q: Can a rule-based system be upgraded directly? Yes—keep the rules as a fallback, replace the core path with BERT intent recognition plus hybrid retrieval, and the handoff scoring mechanism can be deployed independently first.

Q: How should knowledge base weight decay be set? The original article gives 30 days as the starting point for decay; the specific threshold needs to be adjusted according to the frequency of business policy updates.