Intent Recognition Interview Question: Three-Layer Funnel Architecture Breakdown and Answer Template

Why "Just Pass It to the LLM" Is a Red Flag in Interviews

In AI-related job interviews, "how do you implement intent recognition?" is an almost unavoidable question. Many candidates instinctively answer: "Just pass the input directly to a large model for classification." — The moment those words leave their mouth, the interviewer mentally deducts points.

This question looks like it's asking about implementation, but it's really testing your ability to make engineering trade-offs. In real production environments, routing all user utterances through a large model causes latency, cost, and reliability to collapse simultaneously.

There are two classic pitfalls in intent recognition:

• Pure hardcoded rules: Poor generalization — if a user rephrases their request slightly, recognition fails.
• All traffic routed through an LLM: Slow responses, high costs, and unstable outputs.

What interviewers actually want to see is how candidates balance accuracy, latency, and service cost. The core of the answer isn't "which model to use" — it's the engineering mindset of a layered funnel.

Background: The Technical Role of Intent Recognition Intent Recognition is a core subtask of Natural Language Understanding (NLU), originally widely deployed in IVR (Interactive Voice Response) systems and task-oriented dialogue bots. At its core, it maps a user's natural language input to predefined intent categories — such as "check balance" or "book a flight." As conversational AI has proliferated, intent recognition has become a foundational capability for intelligent customer service, voice assistants, and RPA (Robotic Process Automation) systems. In industry, it is typically paired with Slot Filling as the two core modules of task-oriented dialogue — intent determines what the user wants to do, while slots determine which parameters to use in doing it.

Core Idea: Layered Funnel Filtering

The essence of the layered funnel architecture is to let simple requests be consumed quickly at the top layers, intercepting the vast majority of traffic before it reaches the bottom — so that only ambiguous, complex, or edge-case requests sink down to the LLM tool layer.

It's worth noting that the layered funnel architecture isn't an AI-specific invention. It draws from a classic systems design principle: "let cheap operations run first, let expensive operations serve as the fallback." This philosophy originates from the Memory Hierarchy theory in computer science, which Turing Award winner Maurice Wilkes sketched out in 1951: registers → L1/L2 cache → RAM → disk, with speed decreasing and capacity increasing at each level. The design pattern of "putting fast, expensive resources up front and slow, cheap resources at the back" has since been widely applied across engineering disciplines: in databases, the three-layer structure of index cache, query cache, and disk I/O follows the same logic; in CDN networks, edge nodes, regional nodes, and origin servers form a similar layered interception hierarchy. Applying this thinking to NLP intent recognition is a mark of engineering maturity, not a novel invention.

The design principle can be summarized in one sentence: the deeper the layer, the stronger the comprehension — but also the higher the latency and cost. Engineering discipline demands keeping requests at the upper layers as much as possible, only calling on the most expensive resources when truly necessary.

Quantified benefits of the layered funnel: In real industrial deployments, the traffic distribution and cost savings of a three-layer funnel are well-documented. The rule layer typically intercepts 15%–30% of high-frequency fixed commands, with per-request processing cost approaching zero. The context layer handles 60%–75% of conversations, with lightweight model inference costing roughly 1/500 to 1/1000 of a GPT-4-class LLM call. The tool layer processes only 5%–15% of complex requests. Overall, compared to routing all traffic through an LLM, the three-layer architecture can reduce total inference costs by 80%–95%, compressing P99 latency from the second range down to the sub-100ms range.

The architecture consists of three layers: the Rule Layer, the Context Layer, and the Tool Layer. Let's break each one down.

Layer 1: Rule Layer — Fast Upfront Interception

The rule layer handles high-frequency commands with fixed, unambiguous meaning, implemented via keywords, regular expressions, and state machines. Straightforward commands like "open settings," "check balance," or "transfer to agent" require no LLM at all — routing completes in milliseconds.

The advantages are clear: zero compute consumption, extremely fast responses, and high stability, filtering out a large volume of simple requests.

There's one trap to avoid: don't pile up too many rules. The more rules you add, the higher the maintenance cost, and the greater the risk of false positives. The right approach is to retain only high-frequency, long-stable fixed commands.

Layer 2: Context Layer — Handling the Majority of Traffic

In real business scenarios, most conversations cannot be understood from a single sentence alone. When a user says "just give me that one" or "still not working," you need conversation history to make sense of it.

The context layer handles roughly 90% of mainstream traffic, typically using a fine-tuned lightweight small model or semantic vector matching, combined with session state to identify intent.

On lightweight small models: BERT (Bidirectional Encoder Representations from Transformers), introduced by Google in 2018, ushered in a new era of pre-trained language models. To meet industrial deployment needs, researchers developed a series of distilled and compressed variants: DistilBERT reduces the parameter count to 66 million (down from BERT's 110 million), achieving 60% faster inference with less than 3% accuracy loss; TinyBERT uses a two-stage knowledge distillation approach to further compress parameters to 14 million. These models typically have fewer than 100 million parameters and, after fine-tuning on domain data, can reach over 95% intent classification accuracy while keeping single-inference latency to 5–20ms. Compared to large models, inference costs are less than 1% of theirs, and they can be deployed locally without calling external APIs, making them more stable. Semantic vector matching uses tools like FAISS (Facebook AI Similarity Search, Meta's open-source high-performance vector retrieval library) to map user input into high-dimensional vectors and perform nearest-neighbor search against pre-stored intent vectors, supporting millisecond-level retrieval across billions of vectors — well-suited for scenarios with a relatively fixed set of intent categories.

This layer is more flexible than the rule layer, yet faster and cheaper than an LLM, covering the vast majority of everyday conversation scenarios.

The engineering focus of this layer is solid session state management. If the context gets corrupted or out of sync, recognition results will fail immediately — maintaining dialogue state is the key to keeping this layer stable.

Session state management is a well-known engineering challenge in dialogue systems. In distributed service architectures, a user's multi-turn conversation may be routed to different service nodes; if state is not persisted or synchronized, recognition will break down entirely. Common solutions include: using Redis to store session slot data, designing a Dialogue State Tracker based on conversation turns, and assigning a unique session_id to each conversation to track its complete context. State cleanup strategies — timeout durations and turn limits — also require careful design to avoid memory leaks or context contamination.

Intent confidence scores and fallback mechanisms: In a layered funnel implementation, each layer doesn't simply output a binary "recognized / not recognized" — it outputs a confidence score. When confidence falls below a preset threshold, the request drops down to the next layer. Threshold calibration is the core tuning challenge: a threshold set too high causes too many requests to unnecessarily flow to the expensive lower layer; one set too low causes the upper layer to handle low-confidence requests, leading to misrecognition. In practice, thresholds for each layer are typically calibrated through A/B testing and offline evaluation sets, with online monitoring tracking real-time acceptance rates and accuracy at each layer, forming a dynamic optimization loop.

Layer 3: Tool Layer — LLM as the Last Line of Defense

Only requests that have failed recognition in both preceding layers reach the tool layer. This layer uses a large model paired with tool calling as the ultimate fallback — not just identifying intent, but connecting all the way through to business execution.

Once the model understands a complex, cross-domain request, it uses Function Calling or the MCP protocol to match the appropriate tool, fill in parameters, and dispatch directly to business logic for execution — closing the loop from "recognition" to "execution."

The evolution from Function Calling to MCP: Before Function Calling existed, the primary way developers had LLMs invoke external tools was through prompt engineering — crafting a System Prompt that instructed the model to output JSON in a specific format, which external code would then parse and execute. This approach was brittle in terms of format stability and costly to debug. In June 2023, OpenAI formally introduced Function Calling, natively embedding tool invocation capability at the model API layer. Developers can pre-define function signatures and parameter descriptions; the model automatically fills in parameters based on user intent and triggers execution. This dramatically improved the reliability of structured output, marking a paradigm shift from "content generation tool" to "automated execution engine." In 2024, Anthropic released MCP (Model Context Protocol), whose core contribution is standardizing the communication protocol between models and tools — analogous to how the USB interface unified the hardware ecosystem. Any tool conforming to the MCP protocol can be called directly by any MCP-supporting model, breaking the previously fragmented landscape of siloed model ecosystems and advancing interoperability across the AI agent tool ecosystem. Both initiatives share the same goal: upgrading large models from "conversational terminals" to "task execution engines."

The tool layer's strength is its ability to cover any difficult edge case — but it also carries the highest latency and call cost. For this reason, it can only serve as a fallback and must never handle the majority of traffic. Once user volume scales up, costs will spiral out of control.

A Ready-to-Use Interview Answer Template

When faced with a question like "how do you implement intent recognition?", you can structure your answer as follows:

1. Core position: Intent recognition doesn't rely entirely on large models — it uses a three-layer funnel architecture.
2. Rule Layer: Keywords, regex, and state machines handle fixed commands — millisecond-level, low-cost interception of simple requests.
3. Context Layer: Lightweight models (e.g., DistilBERT, TinyBERT) or FAISS semantic vector matching combined with session history handle approximately 90% of conversations, balancing flexibility with performance.
4. Tool Layer: A large model with Function Calling or MCP handles complex requests, closing the full loop from recognition to execution.
5. Summary: Compute resources are allocated differentially by layer — simple requests are resolved at low cost, complex requests only escalate to the LLM — achieving a balance of accuracy, latency, and cost.

Walk through this logic clearly, and the interviewer can immediately tell whether you've genuinely shipped production systems.

Final Thoughts

In intent recognition interview battles, what's being tested is never "how many model names you can recite" — it's your ability to design layered architectures and make sound engineering trade-offs.

Remember the funnel logic: the rule layer handles simple traffic, the context layer carries the bulk of conversations, the tool layer tackles complex requests; the deeper you go, the smarter it gets — and the higher the cost and latency — so the goal is always to resolve requests as high up the stack as possible.

These interview topics almost always come from hard-won lessons in production systems. Understanding the engineering trade-offs underneath — from Memory Hierarchy theory to NLU task design, from lightweight model distillation to protocol standardization — is what makes your interview answers genuinely persuasive and practically actionable.