ToolTurns
A benchmark for tool use in multi-turn dialogue, comprising 2,620 questions, 1,827 tools, and 11,004 API endpoints across 47 categories. It simulates the interaction pattern in which real users progressively clarify their needs.
Timeline (last 90 days)
Microsoft Research and other institutions released ToolTurns, a multi-turn dialogue tool-use benchmark.
ToolTurns contains 2,620 questions, covering 1,827 tools and 11,004 API endpoints across 47 categories.
Each conversation in ToolTurns begins with an initial request that is deliberately incomplete in information, followed by two context-dependent follow-up queries.
The paper evaluated nine language models.
The best agent achieved a resolution rate of 69.6% on individual requests, but its completion rate for full conversations dropped to 39.8%.
Under the ToolTurns evaluation protocol, all evaluated models had lower success rates on follow-up queries than on initial requests.
Conversation quality scores and task success rates may rank models very differently.
ToolTurns comes with an open-source user simulator that progressively reveals hidden context through dialogue and generates follow-up queries.
The authors introduced a checker-fixer process to detect and correct defects in the dialogue turns generated by the simulator.
In blind human evaluations, raters preferred the dialogue turns corrected by the checker-fixer.
1 more timeline events
All Facts (11)
The authors introduced a checker-fixer process to detect and correct defects in the dialogue turns generated by the simulator.
50%UnverifiedIn blind human evaluations, raters preferred the dialogue turns corrected by the checker-fixer.
50%UnverifiedEach conversation in ToolTurns begins with an initial request that is deliberately incomplete in information, followed by two context-dependent follow-up queries.
50%UnverifiedThe paper evaluated nine language models.
50%UnverifiedThe best agent achieved a resolution rate of 69.6% on individual requests, but its completion rate for full conversations dropped to 39.8%.
50%UnverifiedUnder the ToolTurns evaluation protocol, all evaluated models had lower success rates on follow-up queries than on initial requests.
50%UnverifiedConversation quality scores and task success rates may rank models very differently.
50%UnverifiedToolTurns contains 2,620 questions, covering 1,827 tools and 11,004 API endpoints across 47 categories.
50%UnverifiedToolTurns has been open-sourced, and its benchmark and user simulator provide the research community with a reproducible testing platform.
50%UnverifiedToolTurns comes with an open-source user simulator that progressively reveals hidden context through dialogue and generates follow-up queries.
50%UnverifiedMicrosoft Research and other institutions released ToolTurns, a multi-turn dialogue tool-use benchmark.
50%