[KongchangAI]
BenchmarkToolTurns基准

ToolTurns

A benchmark for tool use in multi-turn dialogue, comprising 2,620 questions, 1,827 tools, and 11,004 API endpoints across 47 categories. It simulates the interaction pattern in which real users progressively clarify their needs.

Timeline (last 90 days)

Nov 13

Microsoft Research and other institutions released ToolTurns, a multi-turn dialogue tool-use benchmark.

Unverified50%
Nov 13

ToolTurns contains 2,620 questions, covering 1,827 tools and 11,004 API endpoints across 47 categories.

Unverified50%
Nov 13

Each conversation in ToolTurns begins with an initial request that is deliberately incomplete in information, followed by two context-dependent follow-up queries.

Unverified50%
Nov 13

The paper evaluated nine language models.

Unverified50%
Nov 13

The best agent achieved a resolution rate of 69.6% on individual requests, but its completion rate for full conversations dropped to 39.8%.

Unverified50%
Nov 13

Under the ToolTurns evaluation protocol, all evaluated models had lower success rates on follow-up queries than on initial requests.

Unverified50%
Nov 13

Conversation quality scores and task success rates may rank models very differently.

Unverified50%
Nov 13

ToolTurns comes with an open-source user simulator that progressively reveals hidden context through dialogue and generates follow-up queries.

Unverified50%
Nov 13

The authors introduced a checker-fixer process to detect and correct defects in the dialogue turns generated by the simulator.

Unverified50%
Nov 13

In blind human evaluations, raters preferred the dialogue turns corrected by the checker-fixer.

Unverified50%

1 more timeline events

All Facts (11)

Source Articles