ThinkingBox
An AI agent reliability benchmark published by a Microsoft Research team. It comprises 507 stateful business workflows and evaluates agent stability and reliability in real-world enterprise scenarios through 20 repeated executions combined with backend state verification.
Timeline (last 90 days)
Tool interface design (schema, descriptions, and parameter naming) accounts for 79.9% of root causes of failures, making it the highest-leverage fix point in the technology stack.
The ThinkingBox benchmark, released by Microsoft researchers and Hugging Face, contains 507 real business process tasks, with each task run 20 times.
The ThinkingBox benchmark conducted a total of 121,680 trials, of which 79,853 failed.
Claude Opus 5.5 achieved a perfect 20-out-of-20 score on only 241 of the 507 tasks, for a 20-of-20 reliability of 47.53%.
ThinkingBox performs deterministic evaluation by comparing the final state of the backend database before and after agent operations, and checks for prohibited side effects, without reading conversation logs.
ThinkingBox's tools are exposed as MCP servers, with each run allocated an isolated, fully reset backend.
Kimi-K3's pass@1 is 57.37% and its 20-of-20 is 17.60% (the Hugging Face blog post version reports pass@20 reaching 93.89% with a reliable portion of 13.41%).
The average pass@1 for retail tasks reached 59.52%, while auto insurance stood at only 33.83%, as insurance workflows are harder due to policy rules and precise field requirements.
The ThinkingBox framework is open-sourced on GitHub under the MIT license (microsoft/thinkingbox), with the dataset available on Hugging Face (microsoft/ThinkingBox-Bench).
For production workflows that must succeed every time, the full run-through pass rate should be measured rather than pass@1, because pass@1 systematically overestimates the capability that can be reliably committed to.
13 more timeline events
All Facts (20)
Tool interface design (schema, descriptions, and parameter naming) accounts for 79.9% of root causes of failures, making it the highest-leverage fix point in the technology stack.
50%UnverifiedThe average pass@1 for retail tasks reached 59.52%, while auto insurance stood at only 33.83%, as insurance workflows are harder due to policy rules and precise field requirements.
50%UnverifiedFor production workflows that must succeed every time, the full run-through pass rate should be measured rather than pass@1, because pass@1 systematically overestimates the capability that can be reliably committed to.
50%UnverifiedThe ThinkingBox benchmark, released by Microsoft researchers and Hugging Face, contains 507 real business process tasks, with each task run 20 times.
50%UnverifiedThe ThinkingBox benchmark conducted a total of 121,680 trials, of which 79,853 failed.
50%UnverifiedClaude Opus 5.5 achieved a perfect 20-out-of-20 score on only 241 of the 507 tasks, for a 20-of-20 reliability of 47.53%.
50%UnverifiedThinkingBox performs deterministic evaluation by comparing the final state of the backend database before and after agent operations, and checks for prohibited side effects, without reading conversation logs.
50%UnverifiedThinkingBox's tools are exposed as MCP servers, with each run allocated an isolated, fully reset backend.
50%UnverifiedKimi-K3's pass@1 is 57.37% and its 20-of-20 is 17.60% (the Hugging Face blog post version reports pass@20 reaching 93.89% with a reliable portion of 13.41%).
50%UnverifiedAmong the failures that did not pass the check, 67.24% still terminated cleanly—they invoked state-changing tools and reported no final tool error at the end.
50%UnverifiedScoring with completion-style proxy metrics will classify failures that terminate cleanly but produce incorrect results as successes, which is a flaw in many current agent evaluations.
50%UnverifiedAmong trajectories that terminated cleanly yet actually failed, field value errors accounted for 77.61%, unexpected additional side effects for 43.30%, and missing required side effects for 25.36% (categories overlap).
50%UnverifiedIn ThinkingBox, each task is executed independently 20 times, each starting from a clean backend initial state, requiring each model to run 10,140 trials.
50%UnverifiedThinkingBox's scoring method compares whether the backend state and side effects after task completion are consistent with the required final state; of the 507 tasks, 477 are scored purely based on state, while 30 additionally check a narrow attribute of the final response.
50%UnverifiedThinkingBox distinguishes three metrics: pass@1 (the proportion of successful attempts across all tries), pass@20 (the proportion of tasks that succeed at least once out of 20 attempts), and all-20 (the proportion of tasks that succeed in all 20 attempts).
50%UnverifiedKimi-K3 achieved a pass@20 of 93.89% (476/507) on ThinkingBox, but its all-20 was only 13.41% (68/507).
50%UnverifiedClaude Opus 5 achieved a pass@20 of 79.09% on ThinkingBox, with an all-20 of 47.53% (241 tasks passing consistently).
50%UnverifiedQwen3.8-27B achieves a pass@20 of 89.35% and an all-20 of 7.50% in ThinkingBox.
50%UnverifiedThe rankings produced by pass@20 and all-20 are nearly opposite, revealing that strong capability does not imply high execution stability.
50%UnverifiedThe research team conducted a retrospective ablation analysis across 12 models with a total of 121,680 valid trials, of which 79,853 failed the executability check.
50%