Stop Counting Tokens, Count Deliverables: A New Approach to Evaluating AI Coding Agents

Why token counting is losing its meaning

When a developer hands a coding task to an AI Agent, it does far more than answer once. It first plans, then generates code, runs tests, and repeatedly revises based on failure results. A complete task chain like this may consume 10 to 50 times the tokens of an ordinary conversational prompt.

This data point from the original source material reveals an easily overlooked fact: the more useful the Agent, the more calls it makes, and the higher the token bill naturally is. If a team still uses "token consumption" to measure the efficiency of AI tools, it will reach an absurd conclusion—the more useful the tool, the more "wasteful" it is. What should truly be measured is what it ultimately delivers.

Token is the basic unit of text processed by large language models. Models do not directly read or write "characters" or "words"; instead, they first split the input into tokens (in English, commonly subword fragments; in Chinese, possibly a single character or a smaller fragment), and then calculate context length, inference overhead, and billing in token units. For this reason, the number of tokens determines both how much context can fit into a single call and directly corresponds to the API bill. But a token is only a measure of "input/output volume" and has no necessary relationship with whether a task is completed or whether code is correct—this is precisely the root of its gradual distortion in Agent scenarios.

AI Agent differs from ordinary conversational calls in the key aspect of the "loop." An ordinary call is one prompt and one response; an Agent breaks a goal into multiple steps, autonomously decides the next action (writing code, running commands, reading error reports), and feeds the execution result back into the model as new input, repeating until the task converges. Each loop is a complete model call, so the same task in Agent mode may trigger dozens of calls, multiplying token consumption. Only by understanding this can one understand why applying a single Q&A cost model to an Agent severely underestimates its real overhead and also misunderstands its real value.

From "how much was used" to "what was delivered"

The cost model of traditional AI products is built on single Q&A: one prompt in, one response out, with token volume roughly predictable. But the Agent's working mode is cyclical: planning, coding, testing, and revision together form a closed loop, and each round may trigger new model calls.

This means token consumption is no longer a cost metric but more like a workload metric. The more consumed, the more often it indicates that the Agent is undertaking a more complex task, correcting more errors, and exercising greater autonomy. According to observations from the original source material, this amplification effect is common in real coding tasks and intensifies as Agent capabilities improve.

The evaluation focus therefore needs to shift: from "how many tokens were spent this month" to "how much usable code the Agent delivered this month, how many defects it fixed, and how many labor hours it saved."

What delivery metrics should look like

An actionable evaluation framework can focus on the outcome layer rather than the process layer:

• Task completion rate: of the coding tasks handed to the Agent, how many ultimately produced runnable code
• First-pass rate and revision rounds: how many test-revision cycles the Agent needs to converge
• Human intervention ratio: the extent to which developers must take over or rewrite the Agent's output
• End-to-end delivery time: total time from task assignment to code merge

Together, these metrics answer one question: is the Agent actually shipping things?

First-pass rate (pass@1) comes from a commonly used metric in code generation evaluation: the proportion of code generated by a model for the same problem that passes all tests on the first submission. In the Agent context, it measures the ability to be "usable without rework." It is paired with revision rounds, i.e., how many rounds of "run—observe failure—modify" the Agent goes through before finally converging. Combined, the two reflect the Agent's stability better than looking only at final success: a task that takes ten rounds to fix and a task that passes on the first try are completely different in terms of reliability and cost structure, even if both are ultimately delivered.

End-to-end delivery time (lead time) is a classic metric in software engineering, referring to the total time from when a requirement enters the queue until the change is actually merged and usable. Applying it to Agents avoids looking only at local metrics such as "how fast the model responds"—what truly determines team efficiency is the time of the entire chain from task assignment to code merge, including queuing, testing, review, and rework. This is also why delivery metrics must appear as a group: completion rate shows "whether it got done," pass rate and rounds show "how stably it was done," human intervention shows "how much human effort was saved," and delivery time shows "how much faster the whole thing became."

The real solution to cost anxiety

Treating tokens as a core KPI will induce teams to make short-sighted decisions—limiting the number of Agent calls, compressing context, and preventing it from running tests. These practices appear to reduce the bill but actually sacrifice the most valuable part of the Agent: autonomous error correction and iterative capability.

A more reasonable approach is to treat tokens as elastic resources similar to cloud computing and measure them by unit delivery cost. For example, "comprehensive token cost per completed coding task" rather than absolute consumption. In this way, even if token usage per task rises, as long as delivery efficiency improves faster, the overall economics still improve.

Cost per delivery comes from cost governance practices in cloud computing: evaluate not by "how many resources were purchased" but by "how much was spent per unit of effective output." A common cloud practice is to allocate costs to specific business units and then divide by the number of requests, orders, or tasks produced by that unit to obtain a horizontally comparable efficiency metric. Applied to Agents, this means dividing the comprehensive token cost over a period by the number of successfully delivered tasks in the same period to obtain the "comprehensive token cost per coding task."

The significance of this conversion is that it binds absolute consumption to output. When Agent autonomy increases and per-task calls rise, absolute tokens will inevitably increase, but as long as task completion grows faster, unit delivery cost actually decreases. Conversely, if unit delivery cost continues to rise, it indicates ineffective loops or that task difficulty exceeds the Agent's capability—this is the signal that truly needs investigation.

Practical recommendations for development teams

If your team is introducing AI coding Agents, you can try the following adjustments:

1. On monitoring dashboards, downgrade token usage to a secondary metric and display it alongside task completion count and merged code volume
2. Establish a separate cost accounting dimension for Agent tasks, distinct from ordinary conversational calls
3. Regularly review high-token-consumption tasks to determine whether consumption comes from task complexity or ineffective loops
4. Use A/B comparisons of unit delivery cost under different Agent configurations rather than simply comparing call volume

The core viewpoint of the original source material is worth chewing over repeatedly: as Agents become more capable, the ruler used to measure them must also change. Continuing to stare at the token counter will only lead teams to save money in the wrong places.