Where Does AI Agent Waste Actually Come From? Model Calls or the Orchestration Layer?

The Overlooked Question: Not "What Was Called," But "Why Was It Called"

Most AI Agent observability tools today can tell you one thing: which models your agent called, how many tokens it consumed, and how much it cost. But a Reddit developer, after digging deeper, raised a more fundamental question — why do these calls happen in the first place?

This shift in perspective matters. Knowing the cost is just seeing the outcome; understanding the root cause is what enables real optimization. With that in mind, the developer built an open-source CLI tool called KORA Doctor — released after AUDR — specifically designed to inspect Agent execution traces and identify the true sources of waste from real runtime data.

What Kind of Waste Does KORA Doctor Look For?

KORA Doctor's current detection logic focuses on several categories of execution waste:

• Redundant model calls: The same reasoning is triggered repeatedly
• Missing caching or result reuse: Results that could have been reused are regenerated instead
• Deterministic work delegated to LLMs: Tasks with fixed, predictable answers that don't require a large model are still routed through inference
• Model overkill: Expensive models used for simple tasks
• Excessive planning and orchestration: Planning loops consuming far more resources than necessary

What these detection patterns share is that they go beyond "counting tokens" — they try to answer whether a given call should have happened at all. This is precisely the layer of analysis that current Agent observability tools almost universally lack.

Execution Traces are a prerequisite concept for understanding this detection logic. In Agent systems, a trace is a structured record of every operation that occurs during a complete task execution — including the inputs and outputs of each LLM call, the parameters and return values of tool calls, intermediate reasoning states, and timestamps with latency information. Unlike logs in traditional applications, Agent traces need to capture non-linear, multi-hop decision chains — the same tool may be called multiple times within a single task, and the planning module may repeatedly revise its goals. Tools like OpenTelemetry, LangSmith, and Arize Phoenix have begun standardizing Agent trace collection, but the analysis layer remains thin: most tools can visualize what happened, but lack the ability to reason about whether each step was necessary — and that's exactly the gap KORA Doctor aims to fill.

Feedback from Production: Waste Starts Before the Model Call

What truly shifted the developer's thinking was a piece of feedback from a production environment. A user running Agents in a real business context told him that the biggest source of waste wasn't repeated inference — it was re-fetching the same tool data across multiple execution steps.

This finding is revealing. It suggests that waste can begin before the model is ever called. The orchestration layer itself generates large volumes of unnecessary retrievals, tool calls, and retries — and each redundant operation stacks additional inference costs on top of itself.

In other words, if you're focused solely on "LLM cost optimization," you may be optimizing the wrong thing entirely. The real problem is execution waste that runs through the entire Agent pipeline, with model calls being the most visible — but not necessarily the most expensive — link at the end of that chain.

The Orchestration Layer plays the role of "the brain behind the brain" in multi-step Agent architectures. It decides when to call which tool, how to pass context, whether to retry on failure, and when to terminate a loop. Common orchestration frameworks include LangGraph, AutoGen, CrewAI, and Semantic Kernel. The quality of the orchestration layer's design directly determines the efficiency ceiling of the entire Agent: an orchestrator without state caching may request the exact same data from an external API multiple times within a single task; a planning loop without proper termination logic may continue consuming resources for verification even after the goal has already been achieved. This kind of waste occurs before LLM inference, never shows up on the token bill, yet adds real latency and tool call costs — making it the most easily overlooked layer in any full-stack cost analysis.

Rethinking the Agent Cost Structure

This observation reframes Agent costs from "inference costs" to "full-pipeline execution costs." A typical waste chain might look like this: poorly designed orchestration logic → repeated triggering of tool calls and data fetches → redundant data returned → fed back into the LLM for repeated processing → potentially compounded by failure retries. In this chain, the model call fee is the last item to appear on the bill, but its root cause lives much further upstream.

This also explains why raw token counting can't guide meaningful optimization — it only reveals the "end-of-pipe bill," not the "upstream decisions" that drove it. Reducing real costs requires examining how the orchestration layer manages state, caching, and data reuse.

An Open Question to the Developer Community

The developer explicitly frames this tool as "built around problems exposed by real traces," and in the original post poses an open question to practitioners running production Agents: Where does the biggest waste in your system come from?

• Model calls
• Tool calls
• Retries
• Retrieval
• Planning loops
• Repeated validation
• Something else?

For teams actively deploying Agents, this is a useful checklist to hold up against your own systems. Relevant resources include AUDR (openaudr.dev) and the open-source KORA Doctor (GitHub: Krako-Labs/kora-doctor), which continues to evolve based on production case studies.

Conclusion

The value of this discussion isn't in recommending a particular tool — it's in correcting a cognitive blind spot: Agent waste is not the same as LLM cost. As more systems move from single calls toward complex, multi-step Agents with tools and planning, the real optimization opportunities are often hidden in the orchestration layer's redundant operations, not in simply swapping to a cheaper model. For engineering teams building production-grade Agents, establishing full-pipeline observability that covers retrieval, tool calls, retries, and planning loops is likely to be more impactful than watching the token bill.