Why Are AI Agents So Expensive? A Deep Dive into Hidden Framework Costs

AI Models Are Getting Cheaper — So Why Are Agents More Expensive?

From OpenAI's first reasoning model to today's new generation models, AI inference costs have dropped by two to three orders of magnitude in just two years, while performance has significantly improved. This perfectly illustrates the Jevons Paradox in economics — when a resource becomes cheaper and more accessible, its total consumption actually increases dramatically.

The Jevons Paradox was first proposed by British economist William Stanley Jevons in his 1865 work "The Coal Question." He observed that after James Watt improved the steam engine, coal usage efficiency greatly increased, yet Britain's total coal consumption rose instead of falling. The core mechanism of this paradox is that efficiency improvements lower the per-unit cost of use, thereby stimulating broader application scenarios and larger-scale usage, with total consumption ultimately exceeding the efficiency savings. In the AI field, this phenomenon is quite typical — when GPT-4 era inference costs reached tens of dollars per million tokens, developers carefully calculated every API call; but when costs dropped to a few cents, developers began building agent workflows requiring hundreds of model calls without hesitation, resulting in even higher total spending. As AI becomes readily accessible, more people are willing to use it, spawning even greater computational demands.

Meanwhile, the competitive landscape between open-source and closed-source models is quietly changing. Previously, open-source models typically lagged behind closed-source models by 6 to 18 months in performance, but recent open-source models have become truly competitive. With the rise of Kimi K3, DeepSeek series, GLM, and other models, open-source solutions have become strong alternatives, occupying an increasingly large share on various leaderboards.

The open-source versus closed-source debate in AI models fundamentally reflects two different technical approaches and business logics. Closed-source models (like GPT-4, Claude 3.5) are exclusively controlled by commercial companies for training data, model weights, and inference optimization techniques, monetizing through API fees. Open-source models publish model weights, allowing the community to freely deploy and fine-tune. Since 2024, this landscape has fundamentally shifted — DeepSeek-V3/R1 achieved performance close to GPT-4o at extremely low training costs, Moonshot's Kimi K3 performed impressively on multiple benchmarks, and Zhipu AI's GLM-4 series continues to close the gap. This competitive dynamic forces closed-source vendors to continuously lower prices while also providing agent frameworks with more low-cost model options.

However, a critical misconception exists: the cost of the model itself versus the framework costs wrapping around the model are completely different things. This is the core of the "hidden costs" this article reveals.

Thin Framework vs. Thick Framework: Cost Difference Up to 30x

Most people use AI through chat interfaces, essentially using a "thin framework" — almost directly interacting with the model. But a small group of early adopters use much thicker frameworks, like Claude Code, Cursor, Open Code, Codex, etc. In these tools, you're no longer directly operating the model but working through an external intermediary called a "framework" or "Harness."

The "framework" or "Harness" here technically refers to the orchestration layer wrapped around the large language model. It's not simply a chat interface, but a complete system for tool invocation, context management, and task decomposition. Taking Claude Code as an example, when a user inputs a coding requirement, the framework will: 1) concatenate the user instruction with system prompts; 2) manage tool definitions (such as file read/write, terminal commands, search, etc.); 3) parse the model's returned tool invocation requests; 4) execute operations in the local environment; 5) feed execution results back to the model for the next reasoning round. This "agent loop" architecture is the core design pattern of all current coding agents, and the basic paradigm of agent frameworks like LangChain, CrewAI, etc.

According to Anthropic's data, simply using an agent or multi-agent system can increase costs by 4 to 15 times. Here's an intuitive comparison experiment: a simple prompt asking to create a file with subtitles only consumes about 25 tokens. Based on OpenAI pricing, this prompt costs only about $0.0001 to $0.0005, the model response is about 500 tokens, and altogether it's less than one cent.

But when the same prompt is processed by a thick framework like Claude Code, the entire interaction cost immediately skyrockets to 37 cents — 30 times more expensive from the start.

System Prompts: The "Toll Booth" Entering the Model

The biggest reason for the price surge is the extra overhead at the framework layer. Frameworks like Claude Code, Open Code, Cline, and Cursor add an extra 8,000 to 30,000 tokens to every message sent to the model. It's like a toll booth — you pay to enter the model.

This overhead is called the system prompt. The system prompt is a special input layer in the large language model architecture, positioned before user messages, used to define the model's behavioral boundaries, role settings, and available tools. In the Transformer architecture, system prompts participate in self-attention calculations along with user input, meaning every token in the system prompt affects the model's reasoning path. For coding agents, system prompts typically include: JSON Schema definitions of tool functions, operational specifications and safety constraints, code style guides, error handling processes, etc. Claude Code's system prompts reach tens of thousands of tokens because it has built-in complete definitions for over ten tools including file operations, Git management, terminal commands, browser control, etc. Each additional tool capability requires corresponding system prompt growth, creating a natural contradiction between feature richness and cost.

Every framework contains a certain amount of system prompts, and feature-rich frameworks like Claude Code often have larger system prompts — because they provide many capabilities out of the box, and these capabilities must be described in detail in the system prompts for the model to understand how to interact with the framework.

Cost differences between frameworks are very noticeable:

Framework	Approximate Cost for Same Prompt
Claude Code	About 37 cents
Open Code	About 11 cents
Cline	About 7 cents
Lightweight Framework	About 2.5 cents

Since frameworks like Cline are open-source, developers can even directly read the length and structure of their system prompts in GitHub repositories to understand the underlying mechanics.

Back-and-Forth Loops: The Real Cost Black Hole of Agents

When using frameworks, you must remember: your prompt is not sent to the model just once. In chat applications, question-and-answer exchanges are common; but in agent applications, there's repeated "back-and-forth oscillation" between the model and framework.

Taking file creation as an example, the entire process involves multiple round trips:

1. Model asks framework to check if file exists → Framework checks computer and returns result
2. Model asks framework to create file → Framework executes operation
3. Model asks framework to write content → Framework completes writing

Even for such a simple task, many round trips are involved. For time-consuming tasks like "creating a complete website from scratch," one operation might take 10 to 20 minutes, with countless round trips in between.

The key is that each interaction carries massive system prompts and context information, each round trip "passes through the toll booth," and costs accumulate layer by layer. This is where AI agents truly burn money. More technically speaking, due to the autoregressive generation nature of large language models, each round trip not only needs to resend system prompts but also attach all previous conversation history as context, meaning that as interaction rounds increase, the input token count for each request grows linearly or even super-linearly, forming an ever-expanding cost curve.

Prompt Caching: Vendors' Cost-Saving and Money-Making Technique

Is there a way to reduce this framework cost? The answer is prompt caching. By caching already computed and stored prompt content, even when sending large loads to the model, you can directly reuse previous KV caches, saving time and computational costs.

The underlying mechanism of prompt caching relies on KV Cache (Key-Value Cache) in Transformer models. In standard self-attention mechanisms, the model needs to calculate Key and Value vectors for each token in the input sequence, and these vectors are repeatedly queried during subsequent token generation. For an identical system prompt, recalculating its KV vectors for every request causes huge computational waste. The core idea of KV caching is: when detecting that the prefix portion of a request (usually the system prompt) is completely identical to a previous request, directly reuse the already computed KV vectors, skipping redundant calculations. This not only saves GPU computation (FLOPs in the prefill phase) but also significantly reduces time to first token (TTFT). In actual deployment, this technique typically uses prefix trees (Radix Tree) or hash matching to efficiently manage cache entries, and open-source inference engines like vLLM have built in this functionality.

This feature is especially important for frameworks and has become common across major inference service providers. Visiting Anthropic's pricing page, you can see differentiated pricing for different cache durations. Anthropic even offers expensive caching services for up to one hour, allowing frameworks to save costs on long-duration tasks like building websites or deep research.

Profit Margin of Caching from an Infrastructure Perspective

Let's do the math from a hardware perspective: a single NVIDIA H100 GPU rental costs about $3 per hour and can theoretically fit a full million-token context window into a single H100. The NVIDIA H100 is the current workhorse GPU for AI inference deployment, equipped with 80GB HBM3 high-bandwidth memory per card, with memory bandwidth reaching 3.35TB/s. The "$3 per hour" rental price mentioned reflects current market rates for on-demand GPU rental through platforms like AWS, GCP, or Lambda. The logic of fitting a million-token context window into a single H100 is as follows: at FP16 precision, the KV cache occupation per token depends on model layers and attention head dimensions; for Claude 3.5-level models, the KV cache for one million tokens occupies dozens of GB of memory, which fits exactly into one H100. This means inference service providers can "rent out" the memory space of the same GPU to multiple cache requests, forming a multi-tenant model similar to cloud storage.

Assuming Anthropic can find 12 different locations to store one million contexts, each storing for 5 minutes, then a single H100's hourly revenue from prompt caching would be about $70 — a considerable profit, exceeding hardware rental costs by more than 20 times.

When entering one-hour cache rates, the pricing model changes because long-term GPU reservation is very expensive. Vendors might temporarily transfer unused memory in GPUs to nearby SSD storage (lower cost), then reload it back to GPUs as needed. This strategy is technically called "KV Cache Offloading" or "tiered storage," borrowing the virtual memory design philosophy from operating systems: swapping inactive data from high-speed storage (GPU HBM) to low-speed but large-capacity storage (NVMe SSD). Modern NVMe SSDs have sequential read speeds up to 7GB/s, far below HBM3's 3.35TB/s bandwidth, but at only one percent of the cost. When users initiate new requests within the one-hour cache window, the system preloads the corresponding KV cache from SSD back to GPU memory, a process typically requiring hundreds of milliseconds to seconds of latency, perfectly acceptable for coding agents' background tasks. This is why vendors can offer rates as low as $10 per million input tokens.

Subscription Model: Letting Users Not Worry About Costs

Although these underlying mechanisms are valuable for research, most people using coding agents don't want to calculate expenses item by item. A common method vendors use to retain users is providing a subscription model at the model layer, allowing users to freely use various frameworks within subscription quotas.

Mainstream AI products like ChatGPT, Claude, and Gemini all offer subscriptions. This way users won't be anxious about how much the agent cost mid-work, but can create with peace of mind knowing usage limits refresh periodically. From a business model perspective, subscriptions are a "risk-sharing" mechanism for both vendors and users: vendors take on the risk that users' actual usage might exceed subscription fees, in exchange for stable cash flow and user stickiness; users pay a fixed cost in exchange for psychological freedom when using, avoiding "metering anxiety" in pay-as-you-go models.

Several typical subscription plans:

• Anthropic Pro Plan: $20 per month, bundled access to Claude Code, Claude Chat, and other frameworks, plus free switching between Opus, Sonnet, Haiku, and other models
• ChatGPT Plus: Also supports mixed use of different frameworks and models based on the same subscription metering

For individual users and small to medium teams, subscription models are currently the most practical way to avoid agent cost anxiety. However, it's important to note that subscription models typically have hidden limitations — such as daily or hourly request frequency caps, advanced model usage quotas, etc. When usage exceeds limits, the system may downgrade to lower-performance models or restrict response speed, which is also an important means for vendors to control costs.

Understanding Cost Structure Is Key to Using AI Agents Well

This analysis reveals a fact ignored by most users: the high cost of AI agents mainly comes not from the model itself, but from system prompt overhead at the framework layer and repeated back-and-forth interactions. Understanding this, you can see why the same prompt costs almost nothing in a chat window but costs dozens of cents in a coding agent.

From a more macro perspective, revealing this cost structure also hints at the future competitive focus of the AI industry: when underlying model capabilities become homogeneous and inference costs continue to fall, efficiency optimization at the framework layer will become the key battleground for differentiated competition. Whoever can achieve the same tool invocation capabilities with fewer system prompt tokens, whoever can complete the same tasks with fewer interaction rounds, will establish advantages in cost and user experience.

For different roles, strategies for controlling agent costs also differ:

• Regular users: Choosing subscription models is the best way to avoid cost anxiety
• Developers and enterprises: Understanding framework overhead and making good use of prompt caching are key to controlling agent costs. Additionally, choosing the right framework "thickness" is important — not every task needs Claude Code-level heavyweight frameworks; using lightweight frameworks for simple tasks can save more than ten times the cost
• Open-source community: By optimizing system prompt length and interaction processes, they are continuously driving down extra costs at the framework layer. Directions worth watching include: dynamic system prompts (loading only tool definitions needed for current tasks), context compression techniques (reducing token occupation of conversation history), and more efficient tool invocation protocols

As open-source models rise and caching technology matures, the barrier to using agents is expected to further lower in the future — but the lesson of "hidden costs" is still worth careful study by every AI practitioner and user.

Key Takeaways

• AI model costs are dropping, but agent costs are rising due to the Jevons Paradox — cheaper resources lead to greater total consumption
• The cost difference between thin frameworks (chat interfaces) and thick frameworks (coding agents) can reach 30x for the same prompt
• System prompts are the main culprit of framework costs, adding 8,000 to 30,000 tokens to every model call
• Back-and-forth agent loops multiply costs because each round trip carries full system prompts and context
• Prompt caching technology can significantly reduce costs by reusing computed KV caches, but creates new profit opportunities for vendors
• Subscription models are the most practical way for users to avoid cost anxiety, providing predictable expenses and psychological freedom