Agent Skill Hit Rate Dropping? 4 Methods to Solve the Skill Routing Problem

The Real Problem Behind a Common Interview Question

"When the number of Skills grows too large, how do you maintain an Agent's hit rate?" This question is becoming increasingly common in Agent development interviews. A real-world pitfall: in the early stages of a project with only a dozen or so Skills, the hit rate is excellent. But once the Skill count grows to dozens or even hundreds, the model starts behaving erratically — there's clearly a matching Skill, but it refuses to call it; or it calls one that seems relevant but completely misses the point.

Most people's first reaction is "the model got dumber." But that's not the real issue. When the Skill base grows beyond a certain scale, Skill routing itself transforms from a simple selection problem into a retrieval problem. Understanding this shift is the prerequisite for solving the hit rate decline.

Progressive Disclosure: The Core Mechanism

To understand why optimization efforts often go in the wrong direction, you first need to understand the core design philosophy Anthropic introduced in Agent Skills — Progressive Disclosure.

Progressive Disclosure originated in UX design, referring to presenting complex information in stages to reduce cognitive load. Anthropic brought this concept into Agent architecture to address the capacity constraints of the LLM's context window. Even though today's mainstream models support 100K or even 200K token contexts, loading the full prompts of hundreds of Skills all at once still causes severe "Lost in the Middle" degradation — research shows that a model's attention to information placed in the middle of the context drops significantly, leading to lower routing accuracy. Progressive Disclosure sidesteps this fundamental problem through a two-stage loading approach.

Rather than stuffing every Skill's full content into the context at once, Claude first reads only each Skill's name and description, then decides whether to load that Skill's full content based on that information.

This explains why so many developers optimize in the wrong direction: they spend all their time polishing the Skill's internal Prompt while never touching the Skill's description. In reality, a Skill's name and description are essentially the title and abstract in vector retrieval. If they're vague — like "data analysis assistant," "general office tool," or "code processing expert" — the model simply can't tell them apart when dozens of Skills are laid out side by side. Since the routing stage only exposes the description to the model, description quality directly determines the ceiling on hit rate.

Four Methods to Improve Skill Hit Rate

With that in mind, here are four progressively advanced solutions — from rewriting descriptions to building a retrieval system — covering the full journey from beginner to advanced.

Method 1: Improve Description Distinctiveness

The most fundamental and immediately effective approach is increasing the distinctiveness of Skill descriptions. The key is a shift in framing: don't write "what this Skill is" — write "when to use it."

For example, instead of "responsible for generating PPT," write "use when the user needs to create presentation materials, investor pitch decks, quarterly reviews, or slide decks." Anthropic officially emphasizes that descriptions should include trigger scenarios, not just feature summaries.

The reason is that when routing, the model is fundamentally doing semantic matching. At the technical level, the Transformer's attention mechanism maps user input and each Skill description into a high-dimensional semantic space and determines the best match through similarity. Vague functional labels (like "data assistant") cover too broad an area in semantic space and easily overlap with other Skills; specific trigger scenario descriptions, on the other hand, form clearer boundaries. The closer the scenario description is to how users actually express themselves, the easier it is for the model to establish a correspondence between user requests and Skills — and hit rate naturally improves.

Method 2: Build a Skill Tree (Layered Routing)

Many teams lay out 100+ Skills in a flat structure, forcing the model to judge among hundreds of options every time and significantly increasing the chance of errors.

A better approach is to establish a Skill Tree for layered routing:

• Layer 1: Divide by broad category — e.g., Engineering, Operations, Marketing, Finance
• Layer 2: Subdivide into sub-categories — e.g., Engineering breaks into Code Generation, Code Review, Test Generation

The model first locates the broad category, then precisely matches the specific Skill within it. This reduces the search space by an order of magnitude. This idea mirrors the "hierarchical index" structure in information retrieval — traditional search engines organize documents into multi-level index trees to reduce the computational overhead of linear scanning. With 100 Skills, flat routing requires judging among all 100 options each time (O(N) complexity); with 10 categories of 10 sub-skills each, the first step is 10-choose-1 and the second is another 10-choose-1, reducing complexity to O(log N), shortening error propagation chains, and significantly improving overall accuracy. In fact, many large-scale Agent systems use exactly this kind of layered routing architecture rather than full-set matching in one shot.

Method 3: Add Negative Sample Descriptions (When NOT to Use)

This is the point developers most often overlook. In addition to telling the model "when to use" a Skill, you also need to explicitly tell it "when not to use" it.

For example:

• "Only for generating SQL; not applicable to database design or performance optimization"
• "Only for front-end code generation; does not handle back-end API development"

This design philosophy maps to the concept of decision boundaries in Support Vector Machines (SVM) — a classifier needs not only the features of positive-class samples, but also negative-class samples to define the decision boundary. Without negative sample descriptions, two functionally similar Skills (such as "SQL Generation" and "Database Optimization") have blurry boundaries in semantic space and the model struggles to distinguish them. Explicit "When NOT to Use" statements essentially define a clear "territory" for each Skill. High-quality Skill designs are advised to include this section — it significantly reduces false triggers (those "looks relevant, completely off-target" misfires) and improves overall precision.

Method 4: Recall + Rerank

When the Skill count reaches hundreds or thousands, the first three methods are no longer sufficient on their own. The most effective solution at this scale is to introduce a retrieval mindset: don't let the LLM directly choose from hundreds of Skills.

The correct flow: first use embedding or keyword-based search to recall the Top 10 candidates, then hand those 10 candidates to the LLM for a final decision. This is exactly the core idea behind RAG (Retrieval-Augmented Generation) — coarse filter first, then precise judgment.

From an architecture standpoint, the recall stage typically uses a Bi-Encoder model (e.g., text-embedding-ada-002 or the BGE series) to independently encode the query and each candidate Skill description into vectors, then compute cosine similarity — extremely fast but limited in precision. The reranking stage uses a Cross-Encoder model (e.g., Cohere Rerank or a locally deployed BGE-Reranker) to jointly encode the user query and candidate Skill description together, achieving higher precision at greater computational cost. Applying this mature two-stage architecture to Skill routing effectively transforms the LLM from a "full-set scanner" into a "precision arbiter," maintaining accuracy while dramatically reducing inference cost and latency.

The Paradigm Shift: From Skill Routing to Retrieval System

Looking at all four methods together, a clear evolutionary arc emerges: as the number of Skills grows, the Skill Router gradually becomes a full-fledged retrieval system.

At sufficient scale, Skill routing is no longer just having an LLM answer a multiple-choice question — it becomes a complete "recall → rerank → decide" pipeline. This pipeline is highly analogous to the technical architecture of a search engine: Skill descriptions correspond to document summaries, Skill Tree corresponds to hierarchical indexes, embedding-based recall corresponds to inverted index coarse retrieval, and LLM reranking corresponds to precision scoring. There is already dedicated research on training standalone Skill Routers to address this problem, signaling that it has become an independent research direction within Agent engineering.

For developers, this conceptual shift is especially critical: optimizing Agent hit rate can't stop at prompt engineering — it requires adopting the full methodology of information retrieval, from description quality (corresponding to document summaries), to hierarchical structure (corresponding to index systems), to recall and reranking (corresponding to the retrieval pipeline).

Summary

Returning to the original interview question, a complete answer framework should: first identify the core issue — "too many Skills causes routing to degrade into a retrieval problem" — then walk through the four progressive solutions: improving description distinctiveness, Skill Tree layering, adding negative sample boundaries, and recall + rerank, and finally elevate to the engineering insight that "the Skill Router is fundamentally a retrieval system." This kind of answer demonstrates both theoretical depth and practical, implementable engineering techniques — and fully conveys a deep understanding of Agent systems.

Key Takeaways