Treat the Context Window as a Budget: Token Allocation and Retrieval Strategies

In every model call, the context window is a scarce resource. Money, latency, and attention costs all grow with input tokens. A larger window only raises the ceiling—it doesn't change the essence of the task: every turn must send the minimum set of tokens needed for the model to answer correctly.

Why the Context Window Needs Budget Thinking

Developers often treat the window as a container to fill without limit, which leads to bloated context, rising costs, and degraded answer quality. The core insight is that every input token incurs cost in three ways—money (billed per token), latency (more tokens means slower responses), and model attention (key information gets diluted).

Budget thinking requires doing the math before choosing context. The author uses a hypothetical 8,192-token window to demonstrate this arithmetic: reserve 1,024 for the answer and 512 for message framing and estimation error, leaving an input allowance of 6,656; instructions take 700, tool schemas take 1,000, the current task takes 200, and state takes 500—so 2,400 is already spent before evidence and history, leaving 4,256.

The value of this arithmetic isn't in the specific numbers but in exposing contradictions: for example, six evidence segments of 800 tokens each require 4,800, exceeding the budget by 544; five segments require only 4,000 and fit, but might drop a critical exception.

Context window refers to the maximum number of tokens a model can receive in a single call, covering both input (system prompt, tool definitions, conversation history, retrieved content, etc.) and output. Input exceeding the window is truncated or causes the request to fail. Window sizes vary widely across models, from thousands to hundreds of thousands of tokens. Window size determines the information ceiling for a single call, but doesn't change the engineering principle of "send only the necessary tokens each turn": the larger the window, the more active management is needed—otherwise it's easy to dilute attention and drive up costs by piling on irrelevant content.

Five Budget Lines and Common Pitfalls

According to the article's table, the context budget can be divided into five lines:

• System prompt: grows with policies and operational instructions. Should be precise, versioned, and kept stable where caching is supported.
• Tool schemas: grows with exposed capabilities. Expose tools per task, or let the agent discover them progressively.
• Conversation state: grows with raw turns, summaries, and persistent facts. Compress and retrieve only the state needed for the current decision.
• Retrieval context: grows with number of chunks × chunk size. Rerank and keep the count small and precise.
• Long-term memory: grows with the amount pulled in each turn. Retrieve only a small number of relevant memories, not the entire archive.

Two of these are most commonly underestimated. First, if all registered tools are sent every turn at runtime, irrelevant schemas consume input tokens and interfere with correct tool selection. Second, replaying the entire raw conversation grows without bound unless the application compresses, summarizes, or retrieves on demand.

Tool schema is a structured definition (typically JSON Schema) describing a tool's name, function, parameters, and types, which the model uses to decide whether to call it and how to pass arguments. If every request carries the schemas of all registered tools, it consumes considerable input tokens and may interfere with the model's judgment about which tools are relevant to the current task. Dynamically exposing tools per task, or letting the agent discover tools progressively, can significantly compress this overhead while improving tool-selection accuracy.

Retrieve by Evidence Bundle, Not by Filling with Fragments

The author points out a common mistake: taking the top five chunks just because five fit. The sixth might be the exception to the rules in the first five, and a prompt containing only rules without exceptions will produce confident but wrong answers. The right approach is to rank complete evidence bundles (rules together with exceptions), including the whole bundle or discarding the whole bundle; if the required bundle doesn't fit, report insufficient context rather than answering with partial evidence.

This principle aligns with retrieval experience: a small number of highly relevant chunks beats piling on many. It applies to the entire session—every budget line competes for the same window.

In retrieval-augmented generation (RAG), the system recalls text fragments (chunks) from an external knowledge base based on the query and splices them into context. A common misconception is to take "the top k chunks that fit" by similarity, but relevant fragments may be scattered and incomplete—for example, recalling only rules while missing exceptions. The evidence bundle approach requires treating logically interdependent fragments (rule + exception, condition + consequence) as an inseparable minimum unit, including the whole bundle or discarding it entirely, thereby preventing the model from giving confident but wrong answers based on incomplete evidence.

Practical Recommendations

• Use the provider's precise token counting, including reasoning and output limits; rough counters are only estimates.
• Reserve output and framing overhead first, then allocate input.
• Expose tools per task and avoid full schemas.
• Compress conversation state and retrieve long-term memory on demand.
• Define evidence bundle boundaries (rule + exception), and include or exclude the whole bundle.

The context window is not a warehouse but a budget sheet that must be re-orchestrated every turn.

One-line Summary

The context window is a scarce budget that must be re-orchestrated each turn—calculate precisely across five budget lines and retrieve evidence in complete bundles.

Paragraph Summary

The article argues that the context window is a scarce resource: every input token incurs cost in money, latency, and model attention, and a larger window only raises the ceiling without changing the task essence of "sending the minimum sufficient tokens." The author uses an 8,192-token window to demonstrate budget arithmetic: reserve output and framing overhead first, then allocate input, exposing the contradiction of bundles that don't fit. The context budget can be divided into five lines—system prompt, tool schemas, conversation state, retrieval context, and long-term memory—with tool schemas and conversation state most commonly underestimated. When retrieving, rank complete evidence bundles (rules together with exceptions), including or discarding the whole bundle, rather than taking only the first few chunks that fit, to avoid confident wrong answers.