The Battle of Tool Discovery Architectures: Plan First or Retrieve First?

When large language models are given the ability to call external tools, they typically need to find interfaces relevant to the user's needs from a massive API registry. However, a limited context window simply cannot accommodate all tool definitions at once. So a core engineering question emerges: when it comes to tool discovery, should you plan first and then retrieve, or retrieve first and then plan?

The Mainstream Route: Retrieve-then-Plan (RtP)

The currently dominant approach in the industry is what researchers call Retrieve-then-Plan (RtP). Its logic is straightforward: first embed the user query, then use approximate nearest neighbor search to pull back a batch of candidate tools with the highest semantic similarity from the registry, then inject these tool definitions into the prompt and let the LLM handle subsequent execution planning.

RtP's engineering implementation is relatively mature, and tech stacks like vector databases, embedding models, and ANN indexes can be reused directly. But its implicit assumption deserves scrutiny—does a semantically similar user query necessarily correspond to a functionally appropriate tool? When a user asks "help me analyze this financial report," the closest match in embedding space might be some "text summarization" tool, while the actual solution could be a combination of "table extraction" plus "financial ratio calculation." Similarity does not equal applicability—this is the fundamental tension of the RtP architecture.

Approximate Nearest Neighbor Search (ANN) is the core technology behind vector retrieval. In tool discovery scenarios, each tool definition is encoded by an embedding model into a high-dimensional vector and stored in a vector database. When a user query arrives, it is likewise encoded into a vector, and the ANN algorithm is responsible for quickly finding the several candidate vectors closest to the query vector among massive numbers of vectors. Commonly used ANN indexes include HNSW (Hierarchical Navigable Small World graph), IVF (Inverted File), and PQ (Product Quantization), among others, each making different trade-offs between retrieval speed and recall. Strictly computing the exact nearest neighbors to the query vector requires traversing all vectors, with complexity O(N), and latency is unacceptable at registry scales of tens of thousands; ANN reduces complexity to approximately O(log N) by sacrificing a small amount of recall, making millisecond-level retrieval possible. Understanding ANN is a prerequisite for understanding the engineering feasibility of RtP—precisely because it is fast enough, this "retrieve first, then plan" pipeline can be deployed in production environments.

Semantic similarity in embedding space refers to the distance between two text vectors in vector space (usually measured by cosine similarity or dot product)—the closer the distance, the closer the semantics. But embedding models essentially compress lexical co-occurrence and contextual patterns; they excel at capturing "surface semantics" but struggle to precisely express "functional fitness." For example, "analyze financial report" and "text summarization" may be very close in embedding space because they often co-occur in corpora, but the actual functional need may be a combination of "table extraction" and "ratio calculation." This misalignment between semantic neighbors and functional neighbors is the technical root of the tension in the RtP architecture.

The Alternative: Plan-then-Retrieve (PtR)

To address this problem, research has proposed a counterintuitive alternative paradigm—Plan-then-Retrieve (PtR). In PtR, the LLM no longer passively waits for retrieval results, but instead first generates several "Abstract Tool Stubs" based on user intent. These tool stubs are essentially hypothetical function signatures that describe "what capabilities do I need" rather than "what tools are already available."

An analogy: RtP is like taking a vague requirement description and rummaging through a warehouse for existing finished tools; PtR is like first drawing up a capability blueprint, clearly marking what specifications of functionality are needed, and then taking this blueprint to the warehouse for matching. Tool stubs serve as retrieval queries; they are constructed by the LLM based on its understanding of the task, so the semantic information they carry is closer to the functional requirements themselves, rather than surface-level lexical similarity.

The researchers conducted scaled comparative evaluations of the two architectures across multiple dimensions. The value of this experimental design lies in the fact that it did not stop at validation on a single dataset or a single registry scale, but systematically examined the impact of different conditions on the performance of both architectures. Preliminary conclusions show that PtR demonstrates considerable potential in specific scenarios, especially when the tool registry scale expands and query intent becomes too complex to be summarized by a single semantic vector—the plan-first strategy can bypass the "similarity trap."

Abstract Tool Stubs can be understood as "hypothetical function signatures" generated by an LLM. They do not point to any specific existing tool in the registry, but instead declare in a structured way "what kind of input, output, and capability the current task requires." For example, faced with a financial report analysis need, the LLM might generate two tool stubs: extract_table(document) -> structured_data and compute_ratio(financials, metric) -> float. These stubs are then embedded and encoded, serving as retrieval queries to match real tools in the registry. The value of tool stubs lies in this: their semantics come from a functional decomposition of the task, rather than a literal paraphrase of the user's words, so they can bypass the fundamental dilemma of "what the user says and what the user wants are not the same thing." From an implementation perspective, tool stubs typically require the LLM in the prompt to output in a specific schema (such as function signatures in JSON form), facilitating subsequent parsing and encoding.

The Essential Divergence Between the Two Architectures

The divergence between RtP and PtR ultimately comes down to two answers to the question "what should the retrieval query consist of?"

RtP uses the original user query as the retrieval signal. Its advantage is simplicity and no additional reasoning overhead; but it brings the ambiguity of user language directly into the retrieval process. What the user says and what the user wants are often not the same thing.

PtR uses LLM-generated abstract tool stubs as the retrieval signal. The cost is an extra reasoning call, but what is gained is a retrieval signal closer to functional semantics. In this process, the LLM's role shifts from a "passive tool consumer" to an "active capability definer."

These two paradigms are not mutually exclusive. In scenarios with small tool registries and tool names that are themselves highly descriptive, the extra reasoning overhead of RtP may not be worthwhile; while in production environments with large-scale registries, chaotic tool naming, and highly abstract user queries, PtR's extra planning step may be precisely the key.

Retrieval signal refers to the text content that is actually encoded and used for querying during the vector retrieval stage. In RtP, the retrieval signal equals the original user query; in PtR, the retrieval signal equals the set of tool stubs generated by the LLM. This is the most essential technical difference between the two architectures. The quality of the retrieval signal directly determines the relevance of recalled candidate tools: if the signal itself is vague, ambiguous, or has a lexical gap with the functional description, even the best ANN index cannot retrieve the correct tool. The retrieval signal can be understood as "the question you pose to a search engine"—the closer the question is to the real need, the more reliable the answer. PtR's extra reasoning call is essentially "rewriting the question," translating language from the user's perspective into language from a functional perspective, which is analogous to the idea of "rewrite the query first, then execute" in database query optimization.

Implications for Agent Toolchain Design

This research touches on an easily overlooked link in the LLM Agent tool-calling pipeline. Most current Agent frameworks focus their energy on "how to make good use of retrieved tools," while paying insufficient attention to the upstream decision of "what to retrieve."

From an engineering perspective, the PtR paradigm means the Agent architecture needs to add a "capability abstraction layer"—before tool calling, it must first complete the translation from user intent to capability requirements. This is closer to how human experts solve problems: first figure out what is needed, then go find what is available.

As tool protocols such as MCP drive continued growth in API registry scale, the choice of tool discovery architecture will become increasingly critical. When there are only a few dozen tools in the registry, the retrieval strategy makes little difference; when the number of tools reaches thousands or even tens of thousands, the quality of the retrieval signal directly determines whether the Agent can find the correct tool.

This research does not offer the simple conclusion that "PtR is comprehensively superior to RtP," but rather reveals the respective applicable boundaries of the two architectures. For teams building tool-augmented Agents, the question worth considering is: does your retrieval query truly represent the capability you need?

MCP (Model Context Protocol) is an open protocol proposed by Anthropic and gradually adopted by the industry to standardize the way large language models connect with external tools and data sources. Before MCP, every Agent framework had its own proprietary way of defining tool interfaces, making tools difficult to reuse across frameworks; MCP uses a unified protocol to describe a tool's capabilities, parameters, and calling conventions, enabling tools to be loaded by different Agents like plugins. This standardization directly drove the expansion of tool registry scale—when tools can be reused across ecosystems, the number of candidate tools a single Agent faces rapidly grows from dozens to thousands or even tens of thousands. A change in the order of magnitude of tool count changes the optimal choice of retrieval architecture: at small scale, the noise of ANN retrieval is still tolerable; at large scale, the quality of the retrieval signal becomes the key variable determining Agent success or failure. This is also the era background that makes the PtR paradigm worth attention.

One-line Summary

At large tool registry scales, Plan-then-Retrieve (PtR) may find functionally appropriate tools better than the mainstream Retrieve-then-Plan (RtP).

Paragraph Summary

The article compares two discovery architectures for large language models calling external tools. The mainstream approach, Retrieve-then-Plan (RtP), directly uses the user query for vector retrieval; it is engineering-mature but suffers from the fundamental problem that "semantic similarity does not equal functional applicability." The research proposes a counterintuitive alternative, Plan-then-Retrieve (PtR), in which the LLM first generates abstract tool stubs as retrieval queries, carrying semantics closer to functional needs and thereby bypassing the similarity trap. The article points out that the essential divergence between the two lies in what constitutes the retrieval query: the original user query or an LLM-generated capability description. PtR is not comprehensively superior to RtP: when tool scale is small and naming is standardized, RtP has lower overhead; while in production environments with large-scale registries, chaotic tool naming, and highly abstract queries, PtR's extra planning step may be precisely the key. As protocols such as MCP drive registry expansion, the choice of tool discovery architecture will directly determine Agent success or failure.