SLATE
SLATE is a benchmark for evaluating the long-horizon API selection capabilities of AI agents. It is built on a deterministic e-commerce simulator and contains 100 e-commerce tasks and a library of about 1,000 tools, with end-to-end task success rate as the core evaluation criterion.
Timeline (last 90 days)
The paper arXiv 2611.03982 proposes the SLATE benchmark, specifically designed to evaluate AI agents' long-horizon API selection capabilities.
The SLATE benchmark contains 100 e-commerce tasks and a library of approximately 1,000 tools, built on a deterministic e-commerce simulator.
According to Table 3 of the paper, Gemma-4-12B-it achieves a single-step accuracy of 74.4% and an end-to-end task success rate of only 5.0%.
The paper explicitly states that the SLATE benchmark runs in a deterministic simulator and that its results should not be directly equated with performance in production environments.
The paper argues that for developers, error accumulation and state consistency need to be treated as core design considerations when building API-based agent workflows.
The paper argues that for evaluators, end-to-end task success rate should become a core metric alongside single-step accuracy.
The SLATE benchmark runs in a deterministic simulation environment, where identical inputs yield reproducible outputs, reducing evaluation noise.
The deterministic simulator simplifies the semantic richness and failure modes of real tools, so the paper specifically states that the results cannot be directly extrapolated to production systems.
All Facts (8)
The paper arXiv 2611.03982 proposes the SLATE benchmark, specifically designed to evaluate AI agents' long-horizon API selection capabilities.
50%UnverifiedThe SLATE benchmark contains 100 e-commerce tasks and a library of approximately 1,000 tools, built on a deterministic e-commerce simulator.
50%UnverifiedAccording to Table 3 of the paper, Gemma-4-12B-it achieves a single-step accuracy of 74.4% and an end-to-end task success rate of only 5.0%.
50%UnverifiedThe paper explicitly states that the SLATE benchmark runs in a deterministic simulator and that its results should not be directly equated with performance in production environments.
50%UnverifiedThe paper argues that for developers, error accumulation and state consistency need to be treated as core design considerations when building API-based agent workflows.
50%UnverifiedThe paper argues that for evaluators, end-to-end task success rate should become a core metric alongside single-step accuracy.
50%UnverifiedThe SLATE benchmark runs in a deterministic simulation environment, where identical inputs yield reproducible outputs, reducing evaluation noise.
50%UnverifiedThe deterministic simulator simplifies the semantic richness and failure modes of real tools, so the paper specifically states that the results cannot be directly extrapolated to production systems.
50%