EcoAgent
A benchmark for evaluating AI agent tool retrieval in real-world scenarios. It includes a large collection of real-world tools and query-tool pair data, and covers two evaluation dimensions: semantic retrieval quality and dynamic reliability.
Timeline (last 90 days)
The arXiv paper RATED (2026-10-09) layers a contextual bandit layer on top of semantic tool retrieval, enabling agents to adapt to real-time tool failures without retraining; in simulated interruption scenarios, the wrong-tool selection rate dropped from 86.2% to 2.9%; the accompanying EcoAgent benchmark contains 5172 tools and 210,000 queries.
The research team released the EcoAgent benchmark, comprising 5,172 real-world tools and over 210,000 query-tool pairing data entries.
The EcoAgent benchmark covers two dimensions: offline retrieval and dynamic reliability evaluation.
The research team has open-sourced the RATED/EcoAgent code and dataset, with the GitHub address at Elliotes/EcoAgent.
All Facts (4)
The arXiv paper RATED (2026-10-09) layers a contextual bandit layer on top of semantic tool retrieval, enabling agents to adapt to real-time tool failures without retraining; in simulated interruption scenarios, the wrong-tool selection rate dropped from 86.2% to 2.9%; the accompanying EcoAgent benchmark contains 5172 tools and 210,000 queries.
50%UnverifiedThe research team released the EcoAgent benchmark, comprising 5,172 real-world tools and over 210,000 query-tool pairing data entries.
50%UnverifiedThe EcoAgent benchmark covers two dimensions: offline retrieval and dynamic reliability evaluation.
50%UnverifiedThe research team has open-sourced the RATED/EcoAgent code and dataset, with the GitHub address at Elliotes/EcoAgent.
50%