The True Cost of AI Agents: What Does Each Correct Answer Actually Cost You?

The price of a single model call is predictable. An AI agent that loops repeatedly is not. Every tool call, every retry, every document retrieval adds another billed round trip — and most teams don't know the total until the cloud bill arrives at the end of the month.

A developer submitted a new benchmark to the Kaggle Benchmarking Challenge that reframes the question entirely: not "how much does this model cost?" but "what is the cost per correct answer?" This seemingly simple shift cuts right to the blind spot in agent cost management for production environments.

Benchmark Design: 24 Real Tasks, Deterministic Scoring

The benchmark consists of 24 small, agent-style tasks across three difficulty levels: easy (arithmetic, information extraction, classification), medium (analysis, drafting, prioritization), and hard (multi-hop synthesis, trade-off reasoning, inference). The author emphasizes these are tasks production agents actually perform — not trivia-style knowledge recall.

The key lies in the scoring mechanism: every task has a deterministic checker, and answers are either right or wrong. No LLM judge, no subjective scoring. This design avoids the well-known subjectivity bias of "LLM-as-a-Judge" evaluation and makes results fully reproducible.

Currently, the first five easy tasks are live on Kaggle, running against five hosted models: Gemini 2.5 Flash, Gemini 3.7 Flash, DeepSeek^Ad R1, GPT-OSS 120B, and Qwen3 235B.

A deterministic checker stands in contrast to the "LLM-as-a-Judge" evaluation paradigm. The latter uses a large model to assess another model's output quality — flexible, but prone to score drift depending on the judge model's version or prompt wording, making results hard to reproduce across time or teams. A deterministic checker works like a unit test: acceptable answers or structured rules are defined upfront, and a program compares outputs automatically with a binary pass/fail result. The trade-off is that task design becomes harder — not every real-world task can be reduced to a single correct answer — but the benefit is full auditability. Anyone with the same inputs can independently verify the results, which is especially critical in a cost benchmark.

Kaggle Results: Even Easy Tasks Can Fail

Three observations stand out from the results on the Kaggle live task page.

Only One Model Completed All Tasks

Gemini 2.5 Flash was the only model with results across all five tasks — passing all five and topping the leaderboard with a perfect score of 100.00. Across the three tasks (e3–e5) where all five models participated, 14 out of 15 results passed.

The One Real Failure Is the Most Telling

GPT-OSS 120B — a flagship open-weight model — failed on e5, a one-line information extraction task ("How tall is the Eiffel Tower in meters?") that every other model passed without issue. The author's observation is pointed: easy tasks are supposed to be boring, and they mostly are — but occasionally a model drops one it absolutely should not drop. In production, that dropped task is a real user's request.

When Accuracy Ties, Cost Is Everything

On easy tasks, nearly every modern model is nearly always correct. When accuracy can't differentiate them, the real differentiator is cost per correct answer — which is precisely the number most teams never calculate.

The author also makes an honest disclosure: the three "Error" rows in task e2 were Kaggle platform errors during his first build pattern, not model failures. The runs completed, but the platform failed to register the swapped-in model. He fixed this starting from e3 and chose to leave e2 as-is rather than quietly rebuild it — because a benchmark that can be silently modified is worthless.

Routing Strategy Is What Teams Can Actually Control

The Kaggle tasks test models one at a time, but production agents don't work that way — a router decides which tier handles each request. So the author ran the full 24-task suite locally through four routing strategies using the open-source tool tokenecon, with deterministic scoring:

Strategy	Accuracy	Total Cost	Cost per Correct Answer
Tiered routing	100%	$0.1661	$0.0069
Large model only	100%	$0.2334	$0.0097
Medium model only	67%	$0.0274	$0.0017
Small model only	33%	$0.0036	$0.0004

The results are instructive. Tiered routing — cheap models for easy tasks, escalating to stronger models for hard ones — matches the large model's 100% accuracy while cutting costs by 29%.

But look at the last row: the cheapest tier has the lowest cost per correct answer, yet gets two-thirds of the work wrong. This exposes a trap — cost per correct answer without an accuracy floor is actively misleading. Only after setting a hard requirement like "accuracy ≥ 95%" does tiered routing emerge as the cheapest viable strategy. And that accuracy floor is exactly what most cost dashboards are missing.

The author explicitly notes: this table is a deterministic local simulation, with prices drawn from sample rates in the tokenecon price sheet. The Kaggle task page only reports pass/fail — none of the dollar figures come from Kaggle. This simulation measures routing strategy, which is the part teams can actually control.

Tiered routing works by dispatching requests to models of different capability (and cost) based on task complexity — analogous to cache layers in software architecture. Simple, high-frequency requests go to cheap small models; only when confidence is insufficient or a task is classified as complex does the system "escalate" to a more powerful but more expensive model. Routing can be implemented via rule-based keyword/intent classification, lightweight scoring models, or by having a small model attempt the task first and assign a confidence score to its own output before deciding whether to hand off. tokenecon is the open-source cost simulation tool used here. It lets developers estimate total spend and cost per correct answer across routing strategies using a local price sheet — without making actual API calls — enabling strategy comparison before going live.

Transparency and What's Next

The author appends an honest disclosure list: only 5 easy tasks are currently live on Kaggle; medium and hard tiers already exist in the suite and have been run locally, with the Kaggle versions coming later. Qwen3 235B ran on the 2506 version available in the Kaggle catalog, not the 2507 specified in the original spec. Gemini 3.7 Flash appeared because it was the default placeholder in the Kaggle template task — but since its data reflects real run results, it was kept.

Next up: medium and hard tiers will go live on Kaggle. The author considers those the genuinely interesting territory — easy tasks saturate quickly, and multi-hop reasoning work is where models actually diverge and where routing strategy proves its value. He invites any team running agents in production to fork the framework, plug in their own tasks and prices, and find out what their correct answers are actually costing them.

Why This Framing Matters

The benchmark's core value isn't any specific model ranking — it's the methodology it proposes. When model capabilities converge and accuracy is hard to differentiate, cost efficiency becomes a critical dimension for engineering decisions. More importantly, it calls out the trap of blindly minimizing "cost per correct answer" without first setting an accuracy floor. Only once that floor is in place does cost optimization carry any real meaning.

For teams shipping AI agents to production, this is a sobering reminder: the rate at which model calls burn money needs to be quantified, monitored, and factored into routing decisions — not discovered after the fact when the monthly bill lands.