τ-bench
An evaluation benchmark targeting LLM tool-use and task-planning capabilities, with emphasis on measuring a model's reasoning and execution performance in complex tool-call chains and multi-step planning scenarios.
Timeline (last 90 days)
τ-bench is an agent evaluation benchmark proposed by the Sierra team, used to measure a model's policy compliance in "tool-agent-user" three-party interaction scenarios, covering two domains: airline customer service and retail e-commerce customer service.
In its airline refund tool, τ-bench adopts a two-layer design: the tool description layer writes the policy requiring the model to first fetch the order and explicitly fill in expected_* parameters for reasoning; the server layer validates facts such as cabin class, insurance, and order time, with data coming from the database and server clock, not relying on parameters reported by the model.
τ-bench's criterion is: even if the agent successfully helps the user refund a ticket, as long as the refund action violates policy, the task is still judged as failed, i.e., it measures "task completion under the premise of following the rules".
Each task in τ-bench comes with a domain policy document specifying the conditions for refunds, rebooking, and exchanges; the agent must satisfy the user's requests across multiple turns without violating policy; evaluation metrics include task success rate and policy violation rate.
In the τ-bench retail environment benchmark test (30 test tasks, simulated customers, single run, solvi 0.8.0, gpt-oss-120b as the Agent), without the guard 18 tasks were solved and the environment rejected 10 mutating calls; with the guard 14 tasks were solved and the environment rejected 0 mutating calls.
In the τ-bench retail probe, the action model predicted the environment's rejections with precision and recall of 1.000 on 11,763 held-out transitions, abstaining in only 0.26% of cases.
τ-bench is a benchmark for tool-calling Agents proposed by the Sierra team, simulating retail and airline customer service scenarios, evaluating multi-turn dialogue, tool use and policy compliance, with the environment returning rejections for violating calls.
SWE-bench tests autonomous code repair capability, GAIA tests real-world multi-step task reasoning, and τ-bench tests tool-calling accuracy.
Benchmark environments targeting agent capabilities, such as WebArena, AgentBench, and τ-bench, still rely primarily on sandbox simulations and struggle to capture the real data noise, permission boundaries, latency jitter, and error propagation chains found in production-grade systems.
τ-bench is a benchmark released by Sierra AI for evaluating the reliability of conversational AI agents in real-world business scenarios.
3 more timeline events
All Facts (12)
τ-bench is an agent evaluation benchmark proposed by the Sierra team, used to measure a model's policy compliance in "tool-agent-user" three-party interaction scenarios, covering two domains: airline customer service and retail e-commerce customer service.
50%UnverifiedIn its airline refund tool, τ-bench adopts a two-layer design: the tool description layer writes the policy requiring the model to first fetch the order and explicitly fill in expected_* parameters for reasoning; the server layer validates facts such as cabin class, insurance, and order time, with data coming from the database and server clock, not relying on parameters reported by the model.
50%Unverifiedτ-bench's criterion is: even if the agent successfully helps the user refund a ticket, as long as the refund action violates policy, the task is still judged as failed, i.e., it measures "task completion under the premise of following the rules".
50%UnverifiedEach task in τ-bench comes with a domain policy document specifying the conditions for refunds, rebooking, and exchanges; the agent must satisfy the user's requests across multiple turns without violating policy; evaluation metrics include task success rate and policy violation rate.
50%UnverifiedIn the τ-bench retail probe, the action model predicted the environment's rejections with precision and recall of 1.000 on 11,763 held-out transitions, abstaining in only 0.26% of cases.
50%UnverifiedIn the τ-bench retail environment benchmark test (30 test tasks, simulated customers, single run, solvi 0.8.0, gpt-oss-120b as the Agent), without the guard 18 tasks were solved and the environment rejected 10 mutating calls; with the guard 14 tasks were solved and the environment rejected 0 mutating calls.
50%Unverifiedτ-bench is a benchmark for tool-calling Agents proposed by the Sierra team, simulating retail and airline customer service scenarios, evaluating multi-turn dialogue, tool use and policy compliance, with the environment returning rejections for violating calls.
50%UnverifiedSWE-bench tests autonomous code repair capability, GAIA tests real-world multi-step task reasoning, and τ-bench tests tool-calling accuracy.
50%UnverifiedBenchmark environments targeting agent capabilities, such as WebArena, AgentBench, and τ-bench, still rely primarily on sandbox simulations and struggle to capture the real data noise, permission boundaries, latency jitter, and error propagation chains found in production-grade systems.
50%Unverifiedτ-bench is a benchmark released by Sierra AI for evaluating the reliability of conversational AI agents in real-world business scenarios.
50%Unverifiedτ-bench simulates customer service tasks in industries such as retail and aviation, using task pass rate as the core metric.
50%UnverifiedAgent evaluation benchmark suites include GAIA, AgentBench, and SWE-bench.
50%