Agent Intent Recognition: Engineering Practices Behind the Three-Layer Funnel Architecture

A Classic Question That Gets Candidates Marked Down

"How does an agent perform intent recognition?" This is a high-frequency interview question for LLM-related positions. Bilibili creator Jack pointed out in his sharing that many candidates immediately blurt out "just hand it to the LLM and let it decide," and at that moment the interviewer has already mentally given them a failing mark.

First, we need to clarify a basic concept: Intent Recognition is one of the core tasks of Natural Language Understanding (NLU), with the goal of determining the true purpose behind a user's utterance. Closely paired with it is Slot Filling—extracting the key parameters needed to execute a task from the user's speech. For example, in "Book me a flight to Beijing tomorrow morning," the intent is "book flight," while the slots include the departure time "tomorrow morning" and the destination "Beijing." In traditional dialogue systems, these two tasks were often handled serially by separate modules, but in the LLM era they have gradually been merged into a single model for joint modeling, which reduces error propagation and cuts down on the number of system calls. Understanding this pair of concepts is a prerequisite for grasping the optimization ideas that follow.

Why is the answer "just hand it all to the LLM" unacceptable? Because in real production environments, if every single user input were handed to the LLM to guess the intent, three critical metrics—latency, cost, and stability—would all collapse simultaneously. What this question really tests is never whether you know how to pick a model, but whether you can find an engineering balance among high accuracy, low latency, and controllable cost.

Intent recognition fears two extremes: one is relying entirely on hard-coded rules, which offers zero flexibility—change the phrasing and recognition fails; the other is relying entirely on the LLM to brute-force everything, running every utterance through the model, which is slow, expensive, and unstable. The engineering answer lies in layered design.

The Core Idea: Turn Intent Recognition Into a Funnel

The core of the entire architecture can be summed up in one sentence—turn intent recognition into a funnel. Simple requests are intercepted at the second level by rules, routine requests are handled in bulk by contextual understanding, and only complex, ambiguous requests are handed to the LLM for deep judgment.

This funnel has one clear characteristic: the deeper you go, the smarter it gets, but also the slower and more expensive. Therefore the design goal is to have the vast majority of requests resolved at the upper layers, with only the truly difficult ones sinking down. This trade-off philosophy—"solve the simple cheaply, and only pay the LLM cost for the complex"—is precisely what separates those who have "actually built production systems" from those who "only know how to call APIs."

Layer One: The Rule Layer, Rapidly Intercepting High-Frequency Commands

The rule layer handles requests with clear intent and fixed expressions, relying on the most rudimentary tools: keywords, regular expressions, and state machines. When users say "open settings," "check my balance," or "transfer to a human agent," these commands are unambiguous, and a single rule match can route them within seconds.

These techniques are all long-established and highly mature deterministic methods in NLP engineering. Regular Expressions match text through pattern strings, capturing fixed-structure expressions with extremely low computational overhead; Finite State Machines (FSM) manage the flow direction of multi-turn interactions through predefined state transition graphs. Their greatest advantages are fast response, interpretable results, and zero inference cost, but the trade-off is a lack of semantic generalization—they only recognize the literal text, not the meaning.

But the rule layer has two classic failure points, which are also where interviews can best reveal depth. Both fundamentally stem from the inherent defect of this literal matching:

The Negation Trap

A user inputs "I don't want to check my balance," and pure keyword matching hits "check balance," so the system routes directly to the balance-checking flow, leaving the user baffled. Pure keyword matching has no ability to judge negation semantics—a single "don't" can invalidate an entire rule. In engineering, you must add a lightweight sentence-pattern constraint—directly excluding combinations of high-frequency negation words with keywords so they don't enter the hit list.

The Trap of Swallowed Multiple Intents

A user says "How much data and how many minutes do I have left in my plan?" The rule layer hits the data keyword and jumps directly to data query, swallowing the "remaining minutes" intent. Therefore the rule layer must not stop judging after hitting a single keyword—it must perform a complete data scan to identify all possible high-frequency intent labels. If multiple are hit simultaneously, it marks them as multi-intent and passes them downstream for further processing, never making a single truncation.

Layer Two: The Context Layer, Handling the Bulk of Routine Intents

In real conversations, most intents are not isolated single sentences but must be judged in combination with the preceding context. When a user says "That one, then," what "that one" refers to depends on what was discussed in the previous turn; when a user says "Still not working," which matter isn't working must be reconstructed from the dialogue history.

This layer typically uses a fine-tuned lightweight small model that combines dialogue state and history to perform intent classification and parameter (slot) extraction. Here, "small model" usually refers to pre-trained language models with parameter counts in the hundreds of millions to billions (such as BERT, RoBERTa, or miniaturized open-source LLMs), adapted to specific intent classification tasks through Supervised Fine-Tuning on domain-specific annotated data. Compared to LLMs with hundreds of billions of parameters, small models can keep inference latency within tens of milliseconds and can be deployed privately on an enterprise's own servers, avoiding data leakage and the high cost of per-call billing.

Some may ask: are small models good enough? The answer is yes, but only on the premise that you perform targeted fine-tuning based on your own business scenarios using high-quality annotated data, then privately deploy a lightweight model specialized in intent classification—both accuracy and latency can be held. It is widely believed in the industry that in vertical scenarios, a few thousand to tens of thousands of finely annotated data points combined with a small model can often match or even exceed the intent classification accuracy of general-purpose LLMs, which is exactly why this layer can handle 90% of the traffic.

The engineering key at this layer lies in state management. If dialogue state is poorly maintained and the context becomes muddled, intent recognition will go astray. Therefore the design and maintenance of the dialogue state machine is the core of whether the context layer can stably handle 90% of routine intents.

Layer Three: The Tool Layer, an LLM Fallback for Complex Needs

Only requests that the first two layers can't determine reach the LLM. It is responsible for understanding complex, ambiguous, and even cross-domain needs, then directly selecting which tool to call and how to fill the parameters, routing to the execution action in one step.

Here, "selecting which tool to call and how to fill the parameters" corresponds to the core mechanism in the current Agent field—Tool Use / Function Calling. Mainstream model providers such as OpenAI and Anthropic all offer structured function calling capabilities, allowing the LLM to automatically select pre-registered APIs based on user needs and output parameters conforming to a JSON Schema. This evolves the LLM from a mere "chatterbox" into an executor that "can actually get things done."

This layer is the smartest, able to handle edge cases that neither rules nor small models can crack, but it is also the slowest and most expensive. Precisely because every tool call involves a full round of LLM inference, its latency is usually in the range of hundreds of milliseconds to several seconds, and the per-call cost is far higher than that of rules and small models. Its positioning is "the last line of defense for a small volume of complex requests"—it must never bear the main traffic. Once your user volume grows and you still place it at the first layer, the cost bill will quickly teach you a lesson.

One optimization point worth noting: parameter extraction can be merged with intent classification into the same model, eliminating an extra call and further reducing latency and cost.

An Interview Answer Framework for Agent Intent Recognition

If asked this question, the standard answer framework Jack provides is:

1. Lead with your stance: Intent recognition should not brute-force everything with a single model but should be built as a three-layer "rules—context—tools" funnel.
2. Rule layer: Handles high-frequency fixed commands using keywords plus regex plus negation-pattern constraints; pay attention to multi-intent scanning without truncation.
3. Context layer: Uses a fine-tuned small model to handle 90% of routine intents, balancing speed and flexibility.
4. Tool layer: Uses the LLM as a fallback for complex, ambiguous needs, directly connecting "intent to execution."
5. Highlight the core: Solve the simple cheaply, and only pay the LLM cost for the complex—striking a balance among accuracy, latency, and cost.

When you deliver this, the interviewer will know you're someone who has actually built production systems. Intent recognition isn't a contest of who is better at calling the LLM, but of who better understands layering and trade-offs.

Conclusion

The value of this three-layer funnel architecture essentially lies in pushing engineering constraints upfront into the system design. It reminds us: in the LLM era, capability ceilings certainly matter, but the key to real-world deployment often lies in "when not to use the LLM." For any agent system that must face real traffic, understanding the triangular trade-off among cost, latency, and accuracy determines a project's success or failure far more than pursuing peak single-point model performance.

Key Takeaways