SLATE Benchmark Exposes AI Agent Weakness: 74% Single-Step Accuracy but Only 5% Task Success Rate

AI agents perform impressively on individual tool selection, but once they are required to complete an end-to-end workflow, their success rate drops off a cliff. The SLATE benchmark, proposed in arXiv paper 2611.03982, brings this long-overlooked problem to the forefront with a set of starkly contrasting numbers.

What Is SLATE: Testing Long-Horizon Tool Selection with 100 E-Commerce Tasks

SLATE is a benchmark specifically designed to evaluate AI agents' long-horizon API selection capabilities. It is built on a deterministic e-commerce simulator and includes 100 e-commerce tasks and a library of approximately 1,000 tools.

The key difference from traditional evaluations lies in the judging criteria: SLATE does not check whether the agent selects the right tool at some step, but whether it can complete the entire task from start to finish. This design directly targets the most real pain point when agents are deployed — users want results, not partial credit for the process.

According to the paper, the benchmark runs in a deterministic simulated environment, meaning the same input produces reproducible output, reducing evaluation noise. The authors also caution that results in the simulator should not be directly equated with performance in production environments.

Deterministic simulator means that environment state transitions are entirely determined by predefined rules; the same action sequence necessarily produces the same observations and rewards, with no randomness or external service fluctuations introduced. Such environments are used in agent evaluation to isolate model capability from system noise: if real API calls were allowed, network latency, rate limiting, and return-format drift would all contaminate success-rate statistics. The cost is that the simulator simplifies the semantic richness and failure modes of real tools, so the paper explicitly states that the results cannot be directly extrapolated to production systems.

End-to-end task success rate differs from single-step accuracy in the granularity of evaluation. The former requires the agent to start from an initial state, make tool calls over an arbitrary number of steps, and ultimately bring the environment to the goal state specified by the task; the latter only checks whether the tool selected at a given step matches the annotated correct tool. In long-horizon tasks, single-step accuracy can be viewed as an upper-bound estimate of the per-step correctness probability, while task success rate approximates the product of the per-step correctness probabilities — the more steps, the larger the gap.

The Data Contrast: The Chasm Between 74.4% and 5.0%

Table 3 of the paper presents the performance of two models, and the numerical contrast is striking.

Gemma-4-12B-it achieves a single-step accuracy of 74.4%, but its end-to-end task success rate is only 5.0%. OpenJev-Flash-9B is similar: 74.8% single-step accuracy and 4.7% task success rate.

The two models' single-step accuracies are nearly on par, and their task success rates both hover in the single digits. This shows that the problem is not a special case of one particular model, but a common deficiency in the capability structure of current agents. Selecting the right tool at a single step and chaining dozens or hundreds of steps into a viable path are two qualitatively different abilities.

From the data, there is a significant gap between single-step accuracy and task success rate. In long-horizon tasks, any deviation at any step can accumulate and amplify, ultimately causing the entire workflow to fail.

Why the Task Still Cannot Be Completed Even When Each Step Is Right

The SLATE results point to a fact that is easily masked by benchmark scores: agent capability evaluation cannot rely solely on local accuracy.

The essence of long-horizon tasks is state dependence. Each step's tool selection builds on the previous step's result, and a small early error can throw all subsequent decisions off track. The tool library of 1,000 tools further amplifies the difficulty of selection — the larger the candidate space, the easier it is for tiny single-step errors to be amplified along the chain.

This also explains why the two models have single-step accuracies near 75% but task success rates below 5%. Single-step accuracy measures "is this choice right at this moment," while task success rate measures "can the whole chain converge to the goal." The latter places far higher demands on error recovery, state tracking, and planning consistency than the former.

State dependence and error accumulation are the two key mechanisms for understanding long-horizon task failure. State dependence means that the subsequently available tool set and parameter values depend on the return results of prior calls; the agent must maintain an internal state representation and keep it synchronized with the actual environment state. Once it selects the wrong tool or misparses a return value at some step, the internal state drifts, and subsequent decisions are built on false premises. Error accumulation describes the amplification effect of the single-step error rate along the chain: even with 95% accuracy per step, the end-to-end success rate after 20 chained steps is only about 36%; and in SLATE, where single-step accuracy is about 75% and tasks span dozens of steps, the theoretical product naturally falls in the single-digit range. Together, these two mechanisms explain why improving single-step metrics yields limited gains in end-to-end success rate.

Implications for Agent Evaluation and Deployment

The value of SLATE lies not in ranking models, but in highlighting a missing evaluation dimension. Looking only at single-step metrics makes it easy to overestimate an agent's practical usability.

For developers, this means that when building API-based agent workflows, error accumulation and state consistency must be treated as core design considerations, rather than assuming that single-step correctness automatically leads to end-to-end success. For evaluators, end-to-end task success rate should become a core metric on par with single-step accuracy.

The paper also clearly delineates the boundaries of its conclusions: the results come from a deterministic e-commerce simulator and cannot serve as evidence of equivalent performance in production environments. This is worth keeping in mind for anyone citing these numbers — the 5.0% in the simulator illustrates a structural capability problem, not the real success rate of any specific online system.

Frequently Asked Questions

What does the SLATE benchmark measure?

It evaluates AI agents' API selection capability in long-horizon tasks, using 100 e-commerce tasks and a tool library of approximately 1,000 tools to test whether an agent can complete tasks end to end.

Why is single-step accuracy high but task success rate low?

In long-horizon tasks, each step depends on the previous step's result, and single-step errors accumulate and amplify along the chain. Single-step accuracy reflects only local selection quality, while task success rate reflects whether the entire chain can converge — they measure different capabilities.

Can these numbers represent production environment performance?

No. The paper explicitly states that the benchmark runs in a deterministic e-commerce simulator, and the results should not be regarded as evidence of equivalent production performance.