Why Is Your AI Assistant Consuming So Many Tokens?

Introduction: The Token Mystery Behind a Simple "hi"

You type a simple "hi" into your AI assistant, but the usage meter shows 18,000 tokens consumed. This sounds like a bug, but it actually reveals an easily overlooked reality about how large language models (LLMs) operate: what you see as input is far from everything the model actually processes.

A Reddit user's question sparked widespread discussion: "Why is your AI using so many tokens?" The scenario he described was very specific—simply typing "hi" yet seeing usage spike to 18,000 tokens. This number isn't an isolated case; it points to a pervasive cost black box in AI application development.

The Invisible Drivers of Token Consumption

System Prompts and Context Injection

When you converse with an AI assistant, the model doesn't receive just the "hi" you typed. In most production-grade applications, every request comes with a system prompt that defines the assistant's behavior guidelines, safety rules, output format, and more. This prompt can be hundreds or even thousands of tokens long.

More critically, many applications dynamically inject context: the user's conversation history, relevant document fragments pulled by retrieval-augmented generation (RAG), tool call descriptions, or even summaries of multi-turn conversations. All of this gets concatenated into the request without the user's awareness, collectively forming the input the model actually processes.

According to analysis in the Reddit discussion, the 18,000-token consumption likely comes from a complex prompt template containing extensive system instructions and retrieved documents. The "hi" the user sees is just the tip of the iceberg.

Retrieval-Augmented Generation (RAG) is a technical architecture that combines external knowledge bases with LLMs. Its core workflow: when a user asks a question, the system first retrieves the most relevant fragments (usually in chunks) from a vector database or document store, then concatenates these fragments as context into the prompt and feeds them to the model. RAG's value lies in enabling the model to answer questions involving private data or time-sensitive information without retraining, but the cost is that each request may carry substantial retrieved text. In a typical RAG request, retrieved fragments can occupy thousands of tokens, depending on the number of recalled chunks and the length of each chunk. This is why a seemingly simple conversational request can consume far more tokens than the user input—retrieved documents become a major component of the request payload without the user's awareness.

Tool Definitions and Function Calling Overhead

Modern AI assistants generally support function calling. To let the model know what tools are available and how to invoke them, developers must include JSON Schema descriptions of all tools in every request. If an application integrates a dozen or more tools—search, calculation, database queries—these descriptions alone can consume thousands of tokens.

Moreover, when the model decides to call a tool, the returned results get appended to the context, further driving up token usage. A seemingly simple greeting might trigger the model's tool-availability confirmation process, causing additional token overhead.

JSON Schema is a standard format for describing JSON data structures, widely used in tool-calling scenarios to define function parameter specifications. Developers must write a JSON Schema for each callable tool, declaring the function name, description, parameter types, required fields, and constraints. Modern LLM APIs (such as OpenAI's function calling and Anthropic's tool use) require these schemas to be sent with every request, and the model uses them to decide whether to call a tool and how to construct parameters. The problem is that schema descriptions need to be detailed enough for the model to accurately understand tool purposes, and detailed descriptions mean more tokens. When an application integrates a dozen or more tools, the tool definitions alone can reach thousands of tokens. Additionally, some APIs support control parameters like tool_choice, further increasing request size. Therefore, streamlining tool descriptions and dynamically loading tool sets by scenario are important strategies for controlling token overhead.

Why Is It Hard for Developers to Notice?

The Black Box Effect of Meters

Most AI products show users aggregated usage data, and users cannot see the specific composition of tokens. This opacity creates a cognitive bias: users assume the meter counts their input, when in fact it counts the entire request payload.

The Reddit poster's confusion stems precisely from the meter not distinguishing between "user input" and "system injection." This black box effect has generated numerous similar questions in developer communities, but few products proactively break down token distribution.

The Gray Area of Caching and Duplicate Billing

Some API providers cache repeated context to reduce costs, but caching strategies aren't always transparent. Even with caching, meters may still display usage based on full token counts, or billing rules may differ depending on cache hits. This makes it difficult for developers to accurately estimate actual costs.

How to Diagnose and Optimize Token Consumption

Break Down the Request Payload

If you're developing an AI application, the first step is to log the complete payload of every request. Count tokens separately for system prompts, history messages, retrieved documents, and tool definitions to identify the largest contributors. In many cases, system prompts or retrieved documents are the main sources of consumption.

According to community experience, a well-designed system prompt can be kept under a few hundred tokens, while a sloppy template can balloon to thousands. Regularly reviewing and streamlining prompts is the most cost-effective optimization.

Token counting itself varies across tools. Different models use different tokenizers, and the same text can differ by more than 20% in token count across models. For example, English words are typically split into 1-2 tokens, while each Chinese character may occupy 1-2 tokens. When counting payloads, developers should use the tokenizer corresponding to the target model (such as OpenAI's tiktoken or the Hugging Face tokenizers library) for precise counts, rather than rough estimates based on character or word counts. Additionally, structured content (like JSON and code) typically has higher token density than natural language, so tool schemas and retrieved fragments often consume more tokens than ordinary text of the same length. Understanding the tokenizer's segmentation logic helps make more economical phrasing choices at the prompt design stage.

Inject Context on Demand

Not every turn of conversation requires full context. For simple greetings or confirmation interactions, you can design a lightweight processing path that skips retrieval and tool description injection. Or use conversation summaries instead of raw history to preserve key information while substantially compressing token counts.

Monitoring and Alerting

Set up token usage monitoring in production environments and sample-analyze abnormally high requests. When a request's token count significantly deviates from the baseline, automatically log its payload structure to help pinpoint the source of the problem.

The User Perspective: What Should You Pay Attention To?

If you're just an ordinary user of an AI product and notice abnormal token consumption, pay attention to these points: Does the product provide usage details? Does it distinguish between input and output tokens? Is there documentation about system prompts or context injection? This information helps determine whether consumption is reasonable.

For developers, this Reddit question is a reminder: cost optimization for AI applications starts with visibility into token composition. Between user input and model processing lies a vast gray area, and only proactive breakdown and monitoring can avoid the confusion of "a single hi costing 18,000."

Conclusion

The token consumption mystery is essentially a problem of engineering transparency. The model isn't secretly "eating" your tokens—it's the application's architecture that determines the actual payload of each request. Understanding the overhead of system prompts, context injection, and tool definitions is the first step toward building AI applications with controllable costs.

One-line Summary

A user types "hi" yet consumes 18,000 tokens—the root cause lies in invisible payloads like system prompts, context injection, and tool definitions.

Paragraph Summary

Starting from a Reddit user's confusion over "typing hi consuming 18,000 tokens," this article reveals that the input an LLM actually processes is far more than the single sentence the user sees. Production-grade AI applications attach system prompts, conversation history, RAG-retrieved documents, and JSON Schema descriptions of tool calls to every request—these invisible contents together form the bulk of the request payload. The meter's black box effect prevents users from distinguishing their own input from system injection, and opaque caching strategies further complicate cost estimation. The article recommends that developers log complete payloads, precisely count each component's proportion using the correct tokenizer, inject context on demand, and establish monitoring and alerting, while reminding ordinary users to check whether products provide usage details. The core conclusion: the token consumption mystery is fundamentally a problem of engineering transparency, and cost optimization starts with visibility into token composition.