Tīrtha Bench
A benchmark focused on evaluating the tool-calling capabilities of models. It uses a unified test set and scorer to compare models across dimensions such as accuracy, cost and speed, while accounting for differences between hosting providers.
Timeline (last 90 days)
Tīrtha Bench's core test contains 800 tool-calling tasks, taken from Berkeley's publicly released BFCL dataset, of which 400 are single calls, 200 are multi-tool selection, and 200 are tasks requiring parallel calls.
In addition to the 800 BFCL tasks, Tīrtha Bench adds 120 trap prompts specially written by the authors to induce models to make incorrect tool calls.
Tīrtha Bench currently has 68 models participating in testing, of which 8 entered the same shared comparison of 800 tasks plus 120 traps.
In the Tīrtha Bench shared comparison, the highest accuracy is 95.4%, achieved jointly by Grok 4.7 and Claude Opus 5.5.
Tīrtha Bench's ranking rule is: a model is compared only with models that ran exactly the same tests and were scored in the same way.
The same open model, gpt-oss-120b, scored anywhere from 63.9% to 78.0% on the same 800 tasks depending on the four different hosting providers.
Tīrtha Bench calculated that the dollar cost per 1000 correct calls spans 65 times, from $0.09 to $5.63.
Grok 4.5's performance on Tīrtha Bench was 93.4% accuracy, a 98.3% trap pass rate, 1.85 seconds latency, and $0.0015 per correct answer.
Mikhail Maslenik reproduced Tīrtha Bench's 800 requests on his own machine using a quantized copy of the same model, and scoring with Berkeley's official checker yielded 90.6%, while Tīrtha's own result was 91.0%, a difference of three out of the 800 tasks.
The reproduction process exposed that two entries in the public test set have expected answers that penalize reasonable replies, and that Tīrtha's scorer is a self-rewritten version of Berkeley's; the authors said they will re-score using the official checker.
3 more timeline events
All Facts (13)
The authors acknowledge that the fairness problem in real-work evaluation remains unresolved: a single run of a 45-minute task is just one sample, and how many random seeds are needed for a ranking to be meaningful is still an open question.
50%UnverifiedIn addition to the 800 BFCL tasks, Tīrtha Bench adds 120 trap prompts specially written by the authors to induce models to make incorrect tool calls.
50%UnverifiedTīrtha Bench currently has 68 models participating in testing, of which 8 entered the same shared comparison of 800 tasks plus 120 traps.
50%UnverifiedIn the Tīrtha Bench shared comparison, the highest accuracy is 95.4%, achieved jointly by Grok 4.7 and Claude Opus 5.5.
50%UnverifiedTīrtha Bench's ranking rule is: a model is compared only with models that ran exactly the same tests and were scored in the same way.
50%UnverifiedThe same open model, gpt-oss-120b, scored anywhere from 63.9% to 78.0% on the same 800 tasks depending on the four different hosting providers.
50%UnverifiedTīrtha Bench calculated that the dollar cost per 1000 correct calls spans 65 times, from $0.09 to $5.63.
50%UnverifiedGrok 4.5's performance on Tīrtha Bench was 93.4% accuracy, a 98.3% trap pass rate, 1.85 seconds latency, and $0.0015 per correct answer.
50%UnverifiedTīrtha Bench publishes every request it sends, allowing third parties to reproduce.
50%UnverifiedTīrtha Bench's core test contains 800 tool-calling tasks, taken from Berkeley's publicly released BFCL dataset, of which 400 are single calls, 200 are multi-tool selection, and 200 are tasks requiring parallel calls.
50%UnverifiedThe reproduction process exposed that two entries in the public test set have expected answers that penalize reasonable replies, and that Tīrtha's scorer is a self-rewritten version of Berkeley's; the authors said they will re-score using the official checker.
50%UnverifiedMikhail Maslenik reproduced Tīrtha Bench's 800 requests on his own machine using a quantized copy of the same model, and scoring with Berkeley's official checker yielded 90.6%, while Tīrtha's own result was 91.0%, a difference of three out of the 800 tasks.
50%UnverifiedTīrtha Bench's next phase, Real Work, plans to measure four things: the amount of work before the first real action, context filling, a 45-minute endurance test, and logs that can explain the score.
50%