The Context Window Is a Budget, Not Memory: The Truncation Ledger and Agent Design

Developers often equate the size of the context window with memory capacity, which leads Agents to silently lose critical information during long conversations or complex tool calls. This article is based on an original piece discussing the truncation ledger, reorganizes the FAQ framework, and adds actionable practical advice.

Window Size Is Not Memory Capacity: The Budget Formula

The original article points out that the window is a shared hard ceiling for input and output. The correct budget model is:

budget = window - reserved_output - schemas - system

In other words, the reserved output tokens, the tool schemas billed on every call, and the system prompt must all be deducted from the window first. What remains is the "prompt budget" you can use for conversation history and user input. Many teams only discover the budget is insufficient after designing their Agent, at which point refactoring costs more. Do the subtraction first, then design the Agent.

The original article emphasizes that the window is an "upper limit," not "memory," and certainly not the content you actually send.

Context window refers to the maximum total number of tokens a model can receive in a single call; it holds both the input (system prompt, tool schemas, conversation history, user messages) and the model's generated output. A token is the smallest unit of text the model processes—roughly 4 characters per token in English, while Chinese is denser. The "window as memory" analogy is highly misleading: memory can be paged and expanded, whereas the window is a one-time hard constraint—anything beyond it can only be truncated or rejected. In addition, reserved_output is often overlooked—if you don't leave generation space for the model, replies get cut off mid-way. Tool schemas must be sent with every request each round and are billed, so the more tools you have, the more fixed overhead you carry. Understanding these components is the prerequisite for good budgeting.

HTTP 200 Does Not Mean the Content Is Complete: Truncation Happens in Three Places

FAQ 2 of the original article points out a common misconception: a successful call only means the request was valid, not that the content is complete. Truncation can happen in three places:

1. The client trims history before sending;
2. The framework silently prunes tool return results;
3. The server truncates according to its own policy.

You can only directly control the first two, but all three can be measured externally. The core conclusion is: a 200 response is a transport fact, not a content receipt. If you rely solely on status codes to judge completeness, production issues will be hard to reproduce.

HTTP 200 indicates the server has successfully received and processed the request; it is transport-layer semantics and has nothing to do with the completeness of the information in the response body. In Agent scenarios, the request body may have been trimmed in advance by the client or framework, and the server may also truncate overly long input due to its own length policy—none of which changes the status code. To judge whether content is complete, you need an independent completeness signal, such as marking anchors in the prompt, recording the token count and message count before and after sending, or checksumming and comparing key fields. Treating status codes as content receipts is a common root cause of production issues that are hard to reproduce.

Truncation Strategies Are Not Just About Deleting the Oldest Messages

FAQ 3 of the original article reminds us that different frameworks use different truncation strategies: delete the oldest, delete the middle, prioritize deleting large tool outputs, keep the system prompt plus the latest N tokens, and so on. The most dangerous is "delete the middle"—early decisions disappear while the latest error is retained, so the Agent re-asks questions it has already answered, and the system reports no error at all.

The corrected mental model given in the original article is: know your strategy's name, or you don't know the strategy.

Summaries Drift; Anchors Must Be Preserved Verbatim

FAQ 4 points out that summaries lose exact strings. Identifiers, error codes, column names, and constraints must be preserved as-is. The original article recommends an anchor mechanism:

• Pin decisions with decision: ... lines;
• Pin identifiers with id=...;
• Pin failures with ERROR: ...;
• The rest of the narrative can be summarized.

The principle is: summarize the narrative, never summarize the invariants.

Anchor refers to a key piece of information that must be preserved verbatim in the prompt, usually a decision, identifier, error code, or constraint. Anchors are needed because summarization models tend to rewrite and compress, and exact strings are the most likely to be distorted during rewriting—and once such information is wrong, downstream tool calls or database operations will fail. A common practice is to mark anchors with fixed prefixes (such as decision:, id=, ERROR:) so that truncation scripts and assertions can recognize them with regular expressions and count their survival, turning "whether critical information is lost" from a subjective judgment into a measurable metric.

A Large Window Cannot Replace Retrieval: Treat the Window as a Cache

The conclusion of FAQ 5 is straightforward: for cost and accuracy, a large window is not inherently better than retrieval. Pasting 40 pages of documents means paying for them every round, whereas retrieval only pays in the round where it is needed. The cheaper and more debuggable design is usually: small window + IDs + a data-fetching tool. Treat the window as a cache, not storage; storage should live outside the prompt.

The Truncation Ledger: Quantify What You Will Lose

The original article provides a Python script context_ledger.py that uses heuristic token counting (about 4 characters/token) to simulate a "keep the tail" truncation strategy and reports which anchors are lost. In the example, with a budget of 200 nothing is lost; when the budget drops to 40, decision: use staging-db for the migration dry run is dropped while the latest ERROR is retained—exactly the typical bug of "a wrong assumption stated confidently."

The original article's usage steps are: replace TURNS with real logs; add anchors for your domain (ticket IDs, currency codes, tenant slugs); scan budgets until anchors start disappearing; record that number as the real budget; and wire in assertions so the pipeline fails when an anchor is lost. The original article emphasizes: a ledger without assertions is just a diary.

The core of context_ledger.py is a loop of heuristic counting + keep-the-tail simulation: it estimates the token count of each round's messages at roughly 4 characters/token, accumulates from the newest message backward until it hits the budget ceiling, and treats the excluded earlier messages as truncated losses. The script then scans the retained region with anchor regexes and reports which anchors remain and which have disappeared. The value of this simulation lies in its repeatability and comparability: running the same turn log under different budgets visually shows the tipping point where anchors are lost. However, its token estimates deviate from a real tokenizer, and it only simulates the strategy you declare, so the results should be used as a reference for the budget floor, not as a precise reproduction of server-side behavior.

Strategy Selection and Limitations

The original article provides a strategy selection table: for short sessions, just send everything; for long chats, pin anchors and summarize the narrative; for large documents, retrieve by ID; for tool-intensive loops, prioritize deleting old tool outputs. There are no free options—you can only choose the failure modes you can detect.

At the same time, face the limitations squarely: heuristic token counting may differ from a real tokenizer by 15-20%; the script simulates your strategy, not the provider's hidden policy; it cannot measure attention quality (the model may ignore a line even if it receives it); anchor regexes are just placeholders; and it does not address latency or per-token cost. If you need a verbatim audit record, keep the full transcript in external storage and pass only IDs into the prompt.

A Question Worth Asking

The original article ends with a direct question: which anchor will your Agent lose first? If you can't answer, you only know the window size, not the Agent's behavior. Run the ledger before your next refactor, and commit the turn files alongside the prompt so every change is comparable.

A short script, a real budget, fewer confidently wrong answers.

One-line Summary

Window size is not memory capacity; use a truncation ledger to quantify anchor loss and wire in assertions.

Paragraph Summary

The article corrects the common misconception that the context window equals memory capacity, pointing out that the window is a shared hard ceiling for input and output, and the real prompt budget must deduct reserved output, tool schemas, and the system prompt. Truncation can occur in three places—client, framework, and server—and HTTP 200 only means transport succeeded, not that content is complete. The delete-the-middle strategy is the most dangerous, making early decisions vanish while the latest error is retained with no error reported. Summaries cause exact strings to drift, so invariants such as decisions, IDs, and error codes must be preserved verbatim as anchors. The article advocates treating the window as a cache rather than storage, replacing large-window pasting with retrieval, and introduces the truncation ledger script, which quantifies anchor loss by simulating strategies, emphasizing that assertions are essential to prevent regressions.