RATED Framework: Bringing Reliability-Awareness to AI Agent Tool Retrieval

Why AI Agent Tool Retrieval Needs "Reliability Awareness"

As AI agents increasingly rely on external tools to complete complex tasks, accurate tool retrieval has become a critical factor in ensuring reliable behavior. Yet most existing retrieval methods focus almost exclusively on optimizing semantic relevance, while largely ignoring a factor that is critically important in real-world deployments — whether a tool is actually available at execution time.

A tool that is highly semantically relevant to a query doesn't necessarily respond correctly at runtime. API timeouts, service outages, and interface changes are commonplace in production environments. If a retrieval system keeps routing requests to tools that have effectively failed, the overall reliability of the agent degrades significantly. RATED (Reliability-Aware Tool Retrieval via Execution Dynamics) is a retrieval framework designed specifically to fill this gap.

RATED's Two-Layer Architecture

The core idea behind RATED is to combine robust semantic matching with dynamic reliability adaptation, creating a retrieval mechanism that balances static accuracy with runtime robustness.

Frozen Dual-Encoder + Contrastive Fine-Tuning

At the semantic matching layer, RATED uses a frozen dual-encoder that is fine-tuned via contrastive learning. This design enables the system to build robust semantic correspondences between queries and tools, ensuring solid baseline retrieval quality. The frozen backbone plus lightweight fine-tuning approach also reduces training and maintenance costs.

The dual-encoder architecture is the dominant paradigm in information retrieval. Its core idea is to use two independent encoders to map queries and document/tool descriptions into a shared vector space, then compute a matching score via vector similarity (typically cosine or dot product). This differs from a cross-encoder, which jointly encodes a query and document together for higher precision — but at the cost of slow inference that makes real-time retrieval over large candidate sets impractical. Contrastive fine-tuning trains the model by constructing positive pairs (relevant query-tool pairs) and negative pairs (irrelevant ones), pulling positive pairs closer in the vector space and pushing negative pairs apart, thereby sharpening semantic discriminability. "Freezing the backbone" means the pretrained language model's core weights remain unchanged; only the top projection layers are adjusted. This preserves general semantic capabilities while dramatically reducing annotation data requirements and training compute — a common parameter-efficient fine-tuning strategy in engineering practice.

Lightweight Contextual Bandit Module

The real embodiment of "reliability awareness" is the lightweight contextual bandit module. This module re-ranks candidate tools based on real-time execution feedback. When a tool fails frequently at runtime, the system rapidly lowers the probability of it being selected and instead prioritizes currently available alternatives.

The key advantage of this design is that it adapts quickly to runtime tool failures without expensive model retraining. For agent systems that need to serve online continuously, this translates to lower operational overhead and stronger fault tolerance.

The contextual bandit is a classic problem framework in reinforcement learning, sitting between supervised learning and full sequential decision-making. Its core mechanism: at each step, the agent selects an action (choosing a tool) from the candidate set based on the current "context" (here: query features and historical execution records of each tool), then receives only the immediate reward signal for that action (execution success or failure) — without waiting for a complete conversation trajectory to end. Compared to a multi-armed bandit, the "contextual" version allows the policy to select actions differently based on the specific context, rather than using the same static probabilities for all situations. In RATED, each tool call's execution result (success / timeout / error) is fed back to the bandit module in real time. The module updates the confidence scores of each candidate tool accordingly, fuses them with semantic relevance scores, and re-ranks the list. This explore-exploit trade-off mechanism allows the system to quickly downweight failing tools when new failures emerge, and gradually restore their rankings after recovery — all without modifying the underlying encoder's parameters.

EcoAgent: A Benchmark for Real-World Evaluation

To systematically evaluate retrieval framework performance in realistic conditions, the research team also released the EcoAgent benchmark. It contains 5,172 real-world tools and over 210,000 query-tool pairing samples, covering both offline retrieval quality and dynamic reliability evaluation.

Compared to previous evaluation sets that focused solely on semantic matching, EcoAgent introduces dynamic consideration of tool availability — much closer to the challenges agents actually face in production environments. The creation of such a benchmark is itself an important step toward making tool retrieval research more practically applicable.

Experimental Results: Dramatically Lower Failure Selection Rates

Experimental results demonstrate RATED's advantages across multiple dimensions:

• On real-world queries, RATED achieves a 52.2% relative MRR improvement over ToolLLM's API retriever, indicating meaningful gains in semantic retrieval quality alone.
• Under severe API outage scenarios (30% failure rate), RATED reduces the failure selection rate from 86.2% to just 2.9%. This contrast strikingly illustrates the value of reliability-aware mechanisms — in conditions where nearly nine out of ten requests might hit a failed tool, RATED almost entirely avoids faulty tools.
• In terms of performance overhead, RATED adds less than 1 millisecond of additional latency compared to pure encoder inference — essentially negligible.

In other words, RATED dramatically improves robustness without sacrificing response speed, which is especially important for agent applications with strict real-time requirements.

MRR (Mean Reciprocal Rank) is a common metric for measuring retrieval quality in information retrieval. For a given query, if the correct tool appears at rank k, the reciprocal rank for that query is 1/k; MRR is the mean of reciprocal ranks across all queries. This metric is more sensitive to top-ranked results — a correct tool ranked 1st scores 1.0, ranked 2nd scores only 0.5, and ranked 5th scores just 0.2. Therefore, a 52.2% relative MRR improvement means RATED not only finds the correct tool, but more consistently surfaces it at the top of the candidate list, directly reducing the extra inference overhead agents incur from scanning through multiple results. The Failure Selection Rate is a metric specifically designed by the RATED team for reliability evaluation; it measures the proportion of cases where the retrieval system ranks a failed tool as the top choice among a candidate set containing faulty tools. Together, the two metrics characterize semantic accuracy and runtime robustness as orthogonal dimensions.

Implications for AI Agent Engineering Practice

The RATED research points to a direction worth serious attention: tool retrieval should not be treated as a static semantic matching problem, but as a decision-making process that evolves dynamically with the runtime environment. Incorporating execution feedback into retrieval ranking is a pragmatic path toward improving the reliability of agent systems.

Its approach of using a contextual bandit for online adaptation — rather than retraining the entire model — also aligns with the practical constraints of real-world deployment. For teams building tool-calling AI systems, this line of thinking offers directly actionable insights. The research team has open-sourced both code and dataset (GitHub: Elliotes/EcoAgent) to enable further community validation and extension.