Agent Skill Routing: Retrieval vs. LLM vs. Two-Stage Architecture Compared

When building an Agent, one engineering decision you can't avoid is: as the number of skills grows, should you let the LLM pick the right skill from user requests, or use retrieval for matching first? The question sounds simple, but it's really a three-way balancing act between accuracy, latency, and cost. This article, inspired by an interview question breakdown from a Bilibili creator, clarifies the differences between three routing strategies and the dominant approach in production systems.

The Core Tension: An Impossible Triangle

When asked in an interview "should you use retrieval or let the model choose when there are too many skills?", many people instinctively answer "obviously let the model choose — it's smarter." But that answer falls apart under scrutiny. If you have hundreds of skills, how do you handle the token cost and response latency of stuffing all those descriptions into a prompt?

This question isn't testing who's smarter — it's testing whether you can do the math. Skill routing is fundamentally an impossible triangle of accuracy, latency, and cost. A solid answer must show that you can dynamically choose the right routing architecture based on the scale of skills, rather than stubbornly committing to one approach.

Why Both Extreme Approaches Fall Short

Blindly Trusting the LLM

The first approach is to dump all candidate skill descriptions into the context and rely on the LLM's reasoning to pick the right one. The upside is obvious: accuracy. LLMs understand complex intent and handle ambiguous requests well.

But the downside is fatal. A core design principle in Agent engineering is "progressive disclosure" — its entire purpose is to conserve tokens. Once you have hundreds of skills, their descriptions alone can consume tens of thousands of tokens. Running every request through full LLM inference with that payload isn't just expensive — latency spikes from milliseconds to seconds. This isn't about the model being slow; the math simply doesn't work out at the engineering level.

Side note: "Progressive Disclosure" originally comes from UX design, referring to interfaces that only reveal more information when necessary, reducing cognitive load. In Agent engineering, this idea translates to: rather than injecting full descriptions of all available skills into the initial prompt, introduce relevant context on-demand and in stages. The driving force is the LLM's billing unit — tokens. Taking GPT-4o as an example, input tokens cost a few dollars per million. If an average skill description is 100 tokens and you have 500 skills, that's 50,000 tokens — the cost of that alone may exceed the actual inference per call. More troubling is the "Lost in the Middle" phenomenon: when context grows too long, models pay significantly less attention to information in the middle, meaning that even with all skill descriptions present, hit rate doesn't necessarily improve linearly.

Relying Entirely on Retrieval

The second approach is to vectorize skill descriptions and run similarity matching when a request arrives. The upside: ultra-fast and ultra-cheap — pure math, millisecond-level responses.

But the ceiling is obvious: it only sees surface-level semantic similarity. If a user phrases things differently, or if the intent requires logical reasoning to connect to a skill, vector retrieval will very likely miss or misfire. To borrow a metaphor from the video: it has memory but no reasoning ability.

Side note: Vector retrieval works by mapping text into high-dimensional vectors via an embedding model, then using cosine similarity or dot product to find nearest neighbors. This works beautifully for "semantically similar on the surface" matches but struggles with intent that requires logical leaps. For example, if a user says "my order hasn't arrived," the vector layer might match it to a "shipment tracking" skill. But if a user says "my package got lost and I want a refund," that involves the joint handling of a logistics anomaly and a refund process — pure semantic distance has a hard time routing this accurately to a "claims submission" skill. Additionally, embedding models themselves have domain bias — general-purpose models vary in quality for industry jargon and abbreviations, often requiring domain fine-tuning or hybrid sparse retrieval (e.g., BM25 + Dense Retrieval Hybrid Search) to close semantic blind spots in vertical industry scenarios.

The Industry-Standard Answer: Two-Stage Coarse-Then-Fine Routing

The dominant paradigm for handling large-scale Agent skills in production can be summed up in four words: coarse filter, then fine select.

Stage 1: Retrieval for Coarse Filtering

Facing hundreds or thousands of skills, use low-cost vector retrieval to quickly narrow the field down to a Top-K candidate set. This stage isn't responsible for "getting it right" — it's responsible for "not missing anything," ensuring the correct answer falls within this small pool. Its core value is compressing the search space from massive to single digits at minimal cost.

Side note: The K value in Top-K is one of the most critical hyperparameters in the coarse-filter stage. Too small a K will cut the correct skill out of the candidate set, and no amount of model-layer review can recover it. Too large a K sends more irrelevant skills into the fine-select stage, increasing token overhead and the probability of misselection. In practice, K is typically determined through offline evaluation: on a test set annotated with "correct skills," measure Recall@K at different K values and find the inflection point where the recall curve flattens. Beyond a fixed K, dynamic threshold approaches also exist — setting a minimum similarity score and only keeping results above that threshold, which effectively filters noise in scenarios where skill semantics are highly similar. If the business has a clear skill taxonomy (e.g., "payments," "logistics," "after-sales"), you can also run a lightweight intent classification before retrieval to narrow the search space from all skills to a specific category, then execute vector retrieval — further improving signal-to-noise ratio at the coarse-filter stage.

Stage 2: LLM for Fine Selection

Feed the complete descriptions of these few candidate skills to the LLM and let it make the final decision with full context. Since there are only a handful of candidates, token consumption is manageable and latency stays within acceptable bounds. More importantly, the model's reasoning ability bridges the semantic gap that retrieval can't — handling edge cases where "the words don't match but the intent does."

One-sentence summary: the fast layer handles breadth; the slow layer handles precision. Don't make the LLM hunt for needles in the ocean, and don't make retrieval do reading comprehension. Let each layer do what it does best.

Four Pitfalls You Must Address Before Deployment

Once the principles are clear, interviewers (and real engineering scenarios) will inevitably probe implementation details. These four points are unavoidable:

Leave enough recall headroom. At the coarse-filter stage, it's better to recall a few extra candidates and let the model prune them than to leave the correct option out to save tokens. Once the retrieval layer misses it, no downstream model can rescue it. How you set K must be driven by offline evaluation data — not gut instinct.

Skill description quality is foundational. The effectiveness of vector retrieval ultimately depends on how clearly and distinctly each skill's description is written. Vague descriptions cluster together in vector space, and no amount of engineering can save you. Optimizing routing often starts with going back and rewriting skill descriptions.

Build a routing monitoring system. After launch, you need a hit-rate dashboard, and when something goes wrong, you must be able to trace whether the failure was a retrieval-layer miss or a model-layer misjudgment. Without clear attribution, you can't iterate effectively. A routing system without monitoring is flying blind.

Reject over-engineering. If your Agent only has a dozen skills, don't build a two-stage architecture — just let the model decide directly; it's simpler and more accurate. Two-stage routing is heavy artillery designed for large-scale scenarios. Choosing architecture without considering scale is just showing off.

A Reusable Answer Framework

Pulling the above reasoning into a complete response:

Skill routing isn't a binary choice — it's a scale-dependent tradeoff. At small scale, use the model directly for both accuracy and simplicity. At large scale, adopt a two-stage architecture: retrieval for low-cost coarse filtering to ensure recall, and the LLM for high-precision fine selection to ensure accuracy. Throughout, I'd focus on description quality, recall headroom, and production monitoring to keep the routing system observable and iterable.

This kind of answer demonstrates theoretical depth, engineering specifics, and a pragmatic mindset.

Closing Thoughts

With LLM-related technology choices, the hard part is never the knowledge itself — it's whether you can translate knowledge into problem-solving thinking. Agent routing is no different from RAG optimization, MCP protocol design, or multi-agent collaboration: all of it comes down to the same engineering tradeoff logic — do the math first, then choose, and let scale determine architectural complexity.

Background: Routing Monitoring in Practice

Routing monitoring typically involves two instrumentation layers: the retrieval layer records the Top-K hit list and similarity scores for each request; the fine-select layer records the skill ID and confidence the model ultimately chose. Joining logs from both layers lets you distinguish three failure modes: ① retrieval miss (correct skill never entered Top-K), ② model misjudgment (correct skill was in the candidate set but was eliminated), and ③ missing skill (no skill exists for the request). The fix for each is completely different — retrieval misses call for adjusting K or improving the embedding; model misjudgments call for rewriting the prompt or adding few-shot examples; missing skills need to be fed back to the product team to expand coverage. On the observability tooling side, LLM-native platforms like LangSmith and Phoenix (Arize) natively support trace linking, automatically correlating retrieval and generation spans, significantly reducing the overhead of manual instrumentation.