Tīrtha Bench: The Data Programmers Should Look At When Choosing AI Models

Most model leaderboards compare numbers side by side from different test sets, different prompts, and different graders, then rank directly. This approach may be adequate for general capabilities, but once you need to integrate a model into a coding agent and have it call tools, "which model is the smartest" is no longer the right question. The real questions are: which model makes correct tool calls, how often, how fast, how much does each correct answer cost, and on which hosting provider does it run. Tīrtha Bench attempts to answer this series of questions with a unified test.

One Rule: Same Test, Same Grader

Most leaderboards rank numbers from different sources together; Tīrtha rejects this approach. Its rule is simple: a model is only compared with models that ran the exact same test and were graded in the same way.

The core test contains 800 tool-calling tasks, taken from Berkeley's public BFCL dataset—400 single calls, 200 multi-tool selections, 200 tasks requiring parallel calls—plus 120 trap prompts specifically written by the authors to induce models into making incorrect tool calls. Every model on the shared table completed this full test suite.

Currently 68 models have participated in testing, of which 8 have entered the same shared comparison of 800 tasks + 120 traps. The highest accuracy is 95.4%, achieved jointly by Grok 4.7 and Claude Opus 5.5. Every label on the page comes from a fixed rule viewable on hover, not subjective judgment. The authors' own products are also placed on the same table, with measurements broken down by task type.

BFCL (Berkeley Function Calling Leaderboard) is a function-calling evaluation benchmark launched by UC Berkeley, specifically measuring whether models can correctly generate structured tool/function calls based on natural language instructions. It covers scenarios such as single calls, multi-function selection, parallel calls, and multi-turn conversations, and provides an official checker for programmatic scoring of model outputs. Since tool calling is a core capability of agents, BFCL has become one of the most cited public test sets in this field. The 800 tasks used in this article are taken from BFCL, with additional trap prompts written by the authors, so readers should note when comparing with other leaderboards: BFCL raw scores and this leaderboard's scores are not on the same scale.

Shared Tests Reveal What Leaderboards Can't Show

Hosting Provider Impact Is Severely Underestimated

The same open model, gpt-oss-120b, on the same 800 tasks, scores anywhere from 63.9% to 78.0% depending on four different providers. The gap for Llama 3.3 70B is even more striking: 93.4% on one hosting provider, only 69.5% on another.

This means a common procurement path carries risk: selecting a model from a leaderboard, then finding the cheapest hosting provider to purchase—what you get may not be the performance of the model on the leaderboard at all. For teams integrating models into production, hosting provider selection and model selection are almost equally important decision variables.

Hosting provider (inference provider) refers to cloud vendors or inference platforms that actually deploy and provide model inference services externally, such as Together, Fireworks, Groq, and various self-hosted endpoints. The same open-weight model (e.g., gpt-oss-120b, Llama 3.3 70B) can be served differently by multiple hosting providers, with differences arising from quantization precision (FP8, INT4, etc.), inference engines and batching strategies, context length limits, system prompt injection, and sampling defaults. These engineering details don't appear on model cards but can significantly change tool-calling accuracy and latency. Therefore "model selection" and "hosting provider selection" are actually two independent decisions, and the target hosting provider should be verified separately during procurement.

Look at Cost Per Correct Answer, Not Sticker Price

A model that's cheap but has a high error rate may actually have a higher real cost per correct call. Tīrtha divides the actual cost of each run by the number of correct answers, yielding "dollar cost per 1000 correct calls." The results span 65 times: from $0.09 to $5.63. On the accuracy-vs-cost scatter plot, the upper-left corner is the optimal region, with a yellow band marking the efficiency frontier.

A Single Number Masks Structural Changes Between Versions

DeepSeek^Ad V4.1 Flash's overall score ties with V4 Flash, with the difference within noise range. But broken down, the new version's trap-fall rate drops significantly (trap avoidance rate from 73% to 88%), and parallel call handling is also better. If you only look at the total score, the most valuable part of this upgrade happens to be hidden.

Approval Models Are Also Included in Evaluation

An increasingly common pattern in agent architectures is using a small yes/no model to approve tool calls before they are actually executed. Tīrtha evaluates these "checker" models the same way, including how many incorrect calls they let through—a metric directly related to safety boundaries.

Checker models (guard/approval models) are a front-line gate in the agent pipeline: after the main model generates a tool call but before actual execution, a lightweight yes/no classification model determines whether the call is safe and consistent with user intent, blocking or requiring retry if not. Its value lies in using a low-cost model to provide extra insurance for high-risk actions (deleting files, initiating payments, executing shell commands). But this also introduces new failure modes: checkers may let through incorrect calls (false negatives), or mistakenly block reasonable calls (false positives), both of which need to be measured separately. When evaluating checkers, "how many incorrect calls that should have been blocked were let through" reflects safety boundaries better than overall accuracy.

Every Model Has Its Own Page and Complete Raw Data

Click on any model to enter its dedicated page: ranking compared with models that ran the same test, a one-line interpretation, accuracy breakdown by task type, position on each metric, and all raw results retained—including failed runs, which are marked. Take Grok 4.5 as an example: 93.4% accuracy, 98.3% trap pass rate, 1.85 seconds, $0.0015 per correct answer, each with its ranking.

External Reproduction: Three Tasks Apart

A benchmark that can't be reproduced is just a claim. Tīrtha published every request sent, and someone did reproduce it. Mikhail Maslenik ran these 800 requests on his own machine using quantized copies of the same models, and scored them with Berkeley's official checker, obtaining 90.6%; Tīrtha's own result was 91.0%, a difference of three out of 800 tasks.

Reproduction also exposed problems: two entries in the public test set have expected answers that penalize reasonable responses; additionally, Tīrtha's grader is a self-rewritten version of Berkeley's. The authors state they will re-score using the official checker for direct comparison, and the two problematic entries have been footnoted on the page.

Benchmark replication refers to a third party re-running the same test in an independent environment and comparing results, which is a key step in judging benchmark credibility. Common variables in replication include whether model versions and quantized copies are consistent, whether sampling parameters (temperature, top_p) are the same, whether grader implementations are equivalent, and whether requests are published one by one. Even with everything aligned, the randomness of LLM outputs will bring fluctuations of several tasks, so "three tasks apart" at the 800-task scale is generally considered within noise range. Another value of replication is exposing flaws in the test set itself, such as unreasonable expected answers or deviations between the grader and official implementation—problems that are often only discovered through independent external runs.

Next Step: From Single Calls to Real Work

Single tool calls are the easy part. Real programming work involves long sessions, and what Mikhail pointed out is exactly where models collapse in long sessions: a direct instruction—"connect to the local server and update it"—and one model spent its entire budget reading irrelevant files, never connecting; while a free model completed the connection, update, and pulled logs in a few minutes.

Based on this kind of feedback, the next track, Real Work, will measure four things:

1. Work before the first real action: how many files were opened, how many calls were made, how many tokens and seconds were consumed before the model did what was asked, and whether it ultimately succeeded.
2. Context filling: the same task starting from a clean context, a half-full real session history, and after compression.
3. Endurance test (designed by Mikhail): build a small sync tool in 45 minutes, with the counterpart server changing data format midway, failing every tenth request, and hiding instructions in the data.
4. Logs that explain scores: every step, every retry, every recovery, placed on the same timeline, so low scores become fixable stories.

The authors also admit they haven't solved the fairness problem in real-work evaluation: one run of a 45-minute task is just one sample, and models that get lucky early will look very disciplined. How many random seeds are needed for rankings to be meaningful? This question remains open, and they are soliciting answers from the community and welcome anyone to run those 800 tasks and put their numbers side by side for comparison.

One-line Summary

Tīrtha Bench evaluates the tool-calling capabilities of 68 models with a unified test, revealing hosting provider differences and real costs.

Paragraph Summary

Tīrtha Bench proposes a unified evaluation for coding agent tool-calling scenarios: all models run the exact same 800 BFCL tasks plus 120 trap prompts, compared with the same grader. Currently 68 models participate, with 8 entering the shared comparison, and the highest accuracy is 95.4%. It reveals four phenomena that general leaderboards can't show: the same open model can differ by 24 percentage points in accuracy depending on hosting provider; cost per 1000 correct calls spans 65 times; DeepSeek V4.1 Flash's total score ties with the old version but trap avoidance rate rises from 73% to 88%; and checker models themselves are included in evaluation. An external reproducer ran with the official checker and got 90.6%, only three tasks apart from the authors' 91.0%. The next phase, the Real Work track, will shift toward long-session real work, measuring pre-first-action overhead, context filling, a 45-minute endurance test, and explainable logs.