How AI Agents Actually Work: A Full Breakdown of a Single Tool-Calling Loop

When you ask a product "is it an AI agent," the answer is usually hidden in the few dozen lines of invisible code between your question and its reply. Anthropic's documentation puts it clearly: the model itself never executes anything. It only writes a request; what actually runs it is the code you wrote.

From Chatbot to Agent: Just a Tool List Away

A plain chatbot's logic is monotonous — text in, text out, nothing happens in the world. The arrival of the agent added exactly one thing: alongside your message, you hand the model a "list of tools it's allowed to call."

The key lies in a default-setting detail: Claude decides on every turn whether to call a tool or answer directly. This "per-turn decision" is the root of all the complexity that follows.

This idea has a clear origin. An October 2022 ReAct paper proposed that models should interleave reasoning and action, with actions pulling information from the outside. But what actually determines product behavior isn't the paper — it's the mechanism itself.

What is a "tool"? In the LLM context, a tool isn't a capability the model itself possesses, but an external function that the host application pre-registers with a declared parameter schema. Common tools include search engines, code execution sandboxes, database queries, file read/write, HTTP requests, and so on. The model doesn't run these functions; it only outputs a structured JSON matching the tool's schema, like {"name": "get_weather", "input": {"city": "X"}}. The host application parses that JSON, actually executes the function, then wraps the return value into a tool result block and feeds it back to the model. So the boundary of tools is entirely drawn by the developer: the model can only request tools listed in the schema, and any undeclared operation it cannot trigger. This also explains why the tool list is both a capability list and a permission list.

What is the ReAct paper? The October 2022 ReAct (Reasoning + Acting) paper proposed a prompting paradigm: have the model alternately output "reasoning traces" and "actions," where actions pull factual feedback from the external environment, and that feedback is stitched back into the context for the next round of reasoning. It merged two previously separate things — "chain-of-thought reasoning" and "tool-call decisions" — into a single generation process. ReAct is repeatedly cited because it was the first to systematically describe the interleaved "reason—act—observe" loop structure, which is precisely the skeleton of every agent loop today. Note, however, that ReAct is a prompt-level method, distinct from mechanisms like max_turns in vendor SDKs: it's the latter that determines product behavior.

Reading the Agent Loop Line by Line

Break the whole loop into four swimlanes: the model, the host application (the program wrapping the model), your code, and the tools. The task is simple — compare whether city X or city Y is warmer, then save a note.

• Line 1: The host application sends your question and the tool list to the model.
• Line 2: The model returns not an answer, but a tool use block, with stop reason marked as tool use. This is a request, not an action.
• Line 3: The host application reads the request and calls the tool. Now the tool swimlane lights up while the model swimlane stays dark.
• Line 4: The host application sends the tool output back to the model as a tool result block, along with the existing conversation.

The second round of the loop begins. After seeing the first city's temperature, the model requests the second city's data — the same pattern repeats.

The third round requests saving the note. When the model finally has all the information, the stop reason becomes end turn, and the reply is plain text — that's the stop signal.

Translated into code, it's one sentence: as long as the stop reason is still tool use, keep looping. Anthropic's docs also list other exits: end turn, max tokens, stop sequence, refusal. No need to memorize these — the point is that this is ordinary program control flow, nothing mysterious.

Who Decides the Next Step? The Only Test That Separates Agent from Workflow

Given that the model only makes requests and the loop is executed by code, who stops it? Anthropic describes an agent as "pulling ground truth from the environment (e.g., tool results)" at every step, with stopping conditions like a maximum iteration count. Another SDK calls the corresponding setting max turns.

Now run a swap experiment: same task, same tools, but take decision-making away from the model. Hardcode the flow — step one query city X, step two query city Y, step three save. The model's only job is filling in text within fixed steps.

Anthropic gave these two forms names: a workflow is a system where LLMs and tools are orchestrated through predefined code paths; an agent is a system where the LLM dynamically directs its own process and tool usage. What changes is neither the tools nor the model, but simply "who decides the next step."

Anthropic also lists five workflow patterns: prompt chaining, routing, parallelization, orchestrator workers, evaluator optimizer. The last lets one model generate and another evaluate, which is itself within a loop. So a loop alone doesn't constitute an agent — the real criterion is who's making the decisions.

A brief description of the five workflow patterns:

• prompt chaining: split a task into an ordered series of steps, where each step's model output feeds the next step's input, with a fixed path.
• routing: first a classification step judges which category the input falls into, then dispatches it to the corresponding downstream processing chain — the path is determined by the classification result, but the branches are preset.
• parallelization: split a task into independent subtasks run simultaneously, or run the same task multiple times and aggregate via voting.
• orchestrator workers: an "orchestrator" model decomposes a task into subtasks, dispatches them to several "worker" models to execute, then aggregates the results.
• evaluator optimizer: one model generates an answer, another evaluates and gives feedback, the generator revises accordingly, iterating repeatedly.

All five patterns are predefined code paths. Even when some — like the latter two — contain internal loops, the entry/exit conditions, iteration counts, and role assignments are still hardcoded by code; the model has no power to change the flow mid-way. So they belong to workflows, not agents. This distinction matters for engineering decisions: workflows are more predictable and cost-controllable; agents are more flexible but harder to debug.

Why the Same Loop Is More Expensive and More Error-Prone

See how the trace balloons: each round carries the full previous conversation. And Anthropic's billing docs explain that far more than what you said gets counted toward tokens — tool definitions, tool use blocks, and tool result blocks all count, plus an extra built-in tool-usage prompt is appended.

Anthropic measured a number on its own research system: agents consume roughly 4 times the tokens of chat mode, and multi-agent systems about 15 times. That's data from one vendor, one system — not a universal constant.

What balloons isn't just the bill, but also the model's context window. Anthropic calls this context engineering, defined as "curating and maintaining the optimal set of tokens during inference," with the view that token buildup lowers recall and that the model's attention budget is finite.

Is this just one side's view? Chroma tested 18 models and reported that performance grows increasingly unreliable as input grows — but that's a commercial report, not peer-reviewed, and the tasks were deliberately simplified. The peer-reviewed 2023 paper Lost in the Middle found that models perform best when relevant information is at the beginning or end, and worse when it's in the middle.

To be clear: neither source established "how large the effect is on today's strongest models." Both papers targeted older models.

What is context engineering? Anthropic defines it as "curating and maintaining the optimal set of tokens during inference" — that is, actively managing the context window as a finite and expensive resource rather than cramming in everything possible. Its core assumption is that the model's attention budget is finite, and the more tokens pile up, the lower the recall of key information. Related work includes context compression (summarizing long histories into short text), externalized memory (storing notes outside the window, retrieved on demand), structured truncation of tool results, and giving subtasks their own clean windows. Its difference from prompt engineering lies in the focus: the former on "what to put in the window and how to maintain it," the latter more on "how to phrase this one sentence."

How Teams Actually Cope with Context Pressure

Anthropic offers four tactics: summarize history and restart when nearing the limit; store notes outside the window and pull them back when needed; give a sub-agent a clean window and retrieve only a short summary; and just-in-time retrieval — pulling information at the moment it's needed.

On adding more agents, Anthropic itself has a caution: they suit work that can be split into parallel fragments, but not when every agent needs the same context.

Three Questions to Ask Next Time You Watch an Agent Demo

1. At each step, who chooses what to do next? Your code or the model?
2. What stops the loop?
3. What's the cost of each extra turn of the loop?

Anthropic's advice is to find the simplest solution and add complexity only when necessary, warning that agent autonomy means higher costs and the possibility of compounding errors.

Two questions weren't covered this time: What happens when an agent reads content it shouldn't trust? And what happens when the number of agents grows?

One-line Summary

Models only make requests and code runs the loop — "who decides the next step" is the true dividing line between agent and workflow.

Paragraph Summary

Starting from Anthropic's documentation, the article clears up a common misconception: the model itself never executes anything, it only outputs structured tool requests, and it's the host application that actually calls tools and feeds results back. By breaking down the agent loop into four swimlanes, it shows the loop is essentially one sentence — as long as the stop reason is still tool use, keep going, plain program control flow. From there it draws the only criterion for distinguishing agent from workflow: not whether there's a loop, but who decides the next step — code deciding is workflow, model deciding is agent. It then discusses cost and context pressure: each round carries the full conversation, tool definitions, tool use, and tool result blocks all count toward tokens, Anthropic's own measurement put agent consumption at about 4x chat and multi-agent at about 15x; context bloat lowers recall, and four tactics are offered — summarize-and-restart, external notes, sub-agents, and just-in-time retrieval. Finally, three questions for evaluating any agent demo.