Four-Layer Funnel Architecture: A Design Guide for Production-Grade Agent Intent Routing Systems

Does your Agent truly understand what users are saying? A user says "my card was stolen," and the system triggers the card suspension flow instead of a fraud report. A user mentions both a refund and a points inquiry, but the Agent only handles the first one. The emotional phrase "too slow" hijacks a refund request into the complaint workflow — these are all classic failures of intent routing systems.

This article, based on the second episode of Bilibili's "Creation Workshop" series, systematically breaks down a production-grade four-layer funnel intent routing architecture. From design philosophy and hard-won lessons to deployable code solutions, it will help you build an Agent that truly understands human language.

The Four Death Modes of Intent Routing

Before diving into the architecture, we need to diagnose exactly where things go wrong. Intent routing systems have four typical failure modes, each corresponding to real financial losses.

Mode 1: Intent Confusion. The semantic boundaries between similar intents are too close for a single model to distinguish. In the Banking77 dataset, just card_arrival and card_delivery_estimate account for 30% of misrouted queries — it's not that the model isn't smart, it's that the intents themselves are too similar.

Mode 2: Multi-Intent Blind Spot. Research shows that 18% of user sessions contain two or more intents, but traditional single-label training makes models inherently output only one result, silently swallowing the second instruction.

Mode 3: Emotional Word Hijacking. High-weight sentiment words (like "too slow" or "terrible attitude") distort the attention distribution of semantic routing, incorrectly diverting normal business requests into the complaint workflow.

Mode 4: Routing Avalanche. This is the most insidious failure. The Agent misroutes on the first attempt, retries amplify the error, misrouted requests become duplicate requests, duplicate requests max out the CPU, and CPU overload causes even more misrouting. A/B test data shows that without a circuit breaker, load can spike by 40%-60%. The system returns 200, logs look clean, but 80% of requests have already been misrouted.

All four death modes point to the same root cause: you're treating every request as having the same cost. The fix isn't a bigger model — it's building a funnel.

The Core Mindset Shift: From Classification Problem to Funnel Engineering

Before building anything, you must first correct a critical misconception: intent recognition is not an NLP classification problem.

NLP classification pursues 99% accuracy, but a routing system's goal is joint optimization across three objectives: interception rate × latency × cost. The core question you need to ask is: what kind of requests can be resolved with zero model overhead? What kind of requests are worth spending one to two seconds on an LLM call?

The industry has reached consensus: the small model layer has been eliminated. Four reasons — it's slower than vector routing (small model inference 500ms vs. vector matching 30-100ms), less accurate than LLM FC (classification drift, can't extract slots, can't ask clarifying questions), total cost isn't lower (small model + main model = two calls ≥ single LLM FC call), and maintenance cost is high (training data + model updates + deployment).

So only two paths remain: vector semantic routing is the fast path, LLM FC is the slow path. 80%-85% of queries are resolved on the fast path; only requests that truly require reasoning enter the slow path.

Four-Layer Funnel Architecture in Detail

L0: Pre-compiled Regex + Trie Prefix Tree (<1ms, intercepts 60%-80%)

This is the cheapest layer in the entire system. A user types /weather — no thinking needed, route directly to the weather handler. All slash commands and fixed semantic phrases are matched here instantly.

The implementation uses pre-compiled regular expressions + a Trie prefix tree. But there's a fatal pitfall: regex must be pre-compiled at module load time as module-level variables, not recompiled inside the function on every call.

How costly is this pitfall? The initial implementation placed re.compile inside the routing function. In the test environment with a few dozen requests, everything worked fine. In production with 100,000 daily requests, a single slash command took a full two seconds of CPU time to match — because every request was recompiling the entire regex chain. The fix was a one-line change: move compile to the module top level as a constant. CPU instantly dropped from 95% to 2%.

The scope of a single variable determines the CPU bill for 60% of your site's traffic.

L1: Vector Semantic Routing (30-100ms, intercepts another 15%-25%)

This is currently the most underrated layer. The principle is simple: define 3-20 example sentences for each intent, encode them into embeddings, and each intent forms a clear semantic region in vector space. When a new query comes in, encode it, compute cosine similarity, and it lands in the nearest intent region.

A key detail: the number of example sentences can't be too few. For example, for a "refund" intent, you need to inject variations like "refund," "give me my money back," "I want to return this," "cancel this order for me," "where does the refund go" — 3-20 example sentences to cover enough semantic space.

The performance numbers are impressive: on the CLINC150 benchmark, just 80 fine-tuning samples achieve 85.9% accuracy, and full fine-tuning reaches over 96%. Note: no LLM needed at all — pure mathematical computation, completed in 30-100ms.

For implementation, you can directly use Python's semantic_router library. The core code is just five lines: define the intent route object, configure example sentences, create the router, pass in the encoder, and call it to return the matched intent.

The three-layer cost comparison is crystal clear: L0 regex <1ms, zero cost; L1 vector 30-100ms, $0.0001 per request; L2 LLM 1-2s, $0.002. L1 achieves over 95% accuracy on the Top 20 high-frequency intents — only 1.5 percentage points behind L2, but with 1/30 the latency and 1/30 the cost.

L2: LLM FC + Structured Output (1-2s, handles the final 5%-15%)

Only long-tail intents, multi-intent combinations, and complex queries requiring slot filling are worth the LLM cost. The key at this layer isn't which model to choose, but what format to use to constrain the output.

Never use free-text output for intent names. Free text is an open invitation for hallucinations — the model might output an intent name you never defined, one with a typo, or even one written in a different language. The correct approach is to use Enum + JSON Schema constraints: OpenAI's strict: true + response_format: json_schema mathematically guarantees that results outside the enumerated values are impossible. Anthropic's function calling and Gemini's JSON Schema mode provide the same format guarantees.

L2 also has a critical capability: slot filling. When a user wants to book a flight, the destination, departure city, and date are three slots — missing any one of them means the action can't be executed. The system needs to identify the intent → detect missing slots → ask follow-up questions → receive the filled values → execute. Three boundary conditions must be handled: active slot detection (when a user says "Beijing," they're filling in the departure city from the previous turn, not starting a new intent), cancellation detection ("never mind, forget it" should clear all slots), and context inheritance (when the intent hasn't changed from the previous turn, fill slots directly).

There's also a parameter that's most easily overlooked: Temperature must be set to 0. The default value of 0.7 causes the same sentence to produce three different intents across three calls. You want classification, not diversity. This one-line code change may contribute more to routing stability than L0 + L1 + L2 combined.

SafeNet: The Safety Net (<3% of requests)

The safety net does three things: confidence grading, OOD detection, and circuit breaking.

Confidence is graded into three levels: >0.8 executes directly; 0.6-0.8 executes but is flagged for review, simultaneously sent to a review queue to periodically expand Few-Shot examples; <0.6 does not execute, and the system asks a clarifying question. The core logic: low confidence + blind execution = production incident; low confidence + clarifying question = zero risk.

OOD (Out-of-Domain) Detection: A user says "write me a poem," and all business intent confidence scores fall below 0.3. Without OOD detection, the model is forced to classify into the closest intent — "write a poem" contains the word "request," gets routed to the refund request handler, calls the API, and throws an invalid parameter error. The correct approach: if the maximum confidence score across all intents is < 0.4, classify as OOD and respond directly with "I can only handle refund and booking-related questions."

Circuit Breaker: If the same session hits low confidence (<0.6) three consecutive times, shut down the L2 layer for five requests, keeping only L0 + L1 as degraded service, then re-enable L2 after five requests to verify recovery. This is Netflix Hystrix's circuit breaker pattern applied directly to Agent routing — preventing malicious inputs from maxing out LLM calls and preventing a single bad session from dragging down the entire service.

Observability and Continuous Optimization

Intent routing failures don't manifest as 500 errors — they return 200 with the wrong intent. Traditional health checks can't detect this at all. You need three independent metrics:

1. Misrouting Rate: Sample 100 requests daily, manually label the true intent and compare against routing results. Alert if the daily error rate exceeds 5%.
2. P95 Latency: If it exceeds 300ms, too many requests are penetrating through to L2.
3. Intent Distribution KL Divergence: Monitor day-over-day changes in the Top 10 intents. If it exceeds 20%, the business is changing but the routing hasn't kept up. KL divergence is a silent killer — nobody monitors it until three months later when you discover that 80% of queries for a new product line have been misrouted to old categories.

Continuous optimization relies on a data flywheel: when a user says "you misunderstood," it's automatically labeled as a negative sample; when a user clicks a recommended solution, it's automatically labeled as a positive sample. Negative samples enter the annotation queue to expand the corresponding intent's Few-Shot examples; positive samples are directly added to the Golden Set. Monthly, use the Golden Set (50-100 examples per intent) to recalibrate L1's embeddings and L2's Few-Shot examples. Every routing rule change must pass Golden Set regression testing to prevent fixing one edge case from breaking 90% of common scenarios.

Four Iron Rules of Intent Routing

1. Rules first, models as fallback: Don't spend 1-2 seconds on what can be resolved in 0ms.
2. Use enums for intents, never free text: Mathematical guarantees are ten thousand times more reliable than prompt-based promises.
3. Ask clarifying questions on low confidence, don't guess on high confidence: A clarifying question costs nothing; a misrouted request costs a lost order.
4. Deploy monitoring from day one: Intent drift is a silent killer — you never know when it starts, but you know it will come.

With all four layers combined, latency is only 1/3 of a full-LLM approach, and cost is only 1/4. An Agent that truly understands human language isn't explained into existence through prompts — it's filtered into existence, one layer at a time.