Context Engineering in Practice: Why Keeping Everything Actually Costs Less

When building AI Agents, almost every developer has experienced this scenario: the model is clearly intelligent, but as conversations grow longer, results get progressively worse. Most people's first instinct is "the model got dumber," so they switch to Claude, Codex, or other tools. But a presentation by the Towards AI team at the AI Engineer conference revealed a deeper truth: the problem often isn't the model itself—it's context management.

Even more counterintuitive, their extensive testing revealed that summarization and compaction of context can actually be more expensive while degrading quality—and "doing nothing, keeping all history" turned out to be the optimal solution. Behind this counterintuitive conclusion lies an economic revolution driven by caching mechanisms.

Context Rot: The Real Culprit Behind Agent Performance Degradation

The Towards AI team set five core requirements for their AI Tutor: answers must be based on course content rather than the model's own knowledge, anchored to the current lesson, support long debugging sessions, handle code, and maintain low latency. All these requirements essentially point to a single technical challenge—context engineering.

They identified two fundamental problems with models. First, the context window is finite. The context window refers to the maximum number of tokens a large language model can process in a single inference—tokens are the basic units models use to process text, with one English word typically corresponding to 1-2 tokens and Chinese characters typically corresponding to 1-2 tokens. Although model windows have expanded from the early 4K to million-level capacities, research shows models exhibit the "Lost in the Middle" phenomenon, where attention to information in the middle of the window drops significantly. From system prompts, course content, and tool definitions to chat history, everything is crammed into the same limited space—the more you pile in, the worse the results and the higher the costs. Second, models are stateless. This is a fundamental property of the Transformer architecture: each API call is an independent inference process, models have no built-in memory mechanism, and all information that needs to be "remembered" must be explicitly placed in the input. When a student reopens the assistant, the model knows nothing about anything that happened before.

After analysis, the team found that the biggest bottleneck when scaling context was old tool outputs—including text chunks from previous retrievals, all tool call-result pairs, and file search records. These are not only expensive but also trigger so-called "Context Rot."

The deep mechanism of context rot stems from limitations of the Attention Mechanism. Transformer models compute the association weight between each token and all other tokens through self-attention, theoretically capable of capturing dependencies at any distance. But in practice, when context becomes too long, attention weights get diluted by large amounts of irrelevant information, making it difficult for the model to precisely locate key information. This is similar to a human being asked about a specific detail on page 37 after reading a 600-page book—the information was "seen," but retrieval efficiency drops dramatically. Due to limitations in how large language models are trained, they aren't good at grasping the big picture in ultra-long contexts; they passively absorb massive amounts of facts, causing quality to decline as length increases.

The Complete Arsenal of Context Compression Techniques

To manage context, the team systematically cataloged multiple techniques. The most basic are "cheap tools" that don't rely on LLMs: truncating abnormally long tool outputs (keeping only the beginning and end with a truncation marker), using sliding windows to keep only the most recent N turns of conversation, and directly clearing most output for specific tools.

Advanced approaches involve "spending tokens to save more tokens"—using language models to compress context. The most effective techniques in testing included:

• Selective Retention: Having the model decide what to keep or discard based on conversation direction
• Summarization: Continuously summarizing previous turns
• Delta Summarization: Used by Claude Code when spawning sub-agents, continuously updating a dynamic summary

You might not have noticed, but the team didn't adopt sub-agents because their single main-agent architecture already worked well, avoiding unnecessary complexity.

Additionally, there are "Offloading" strategies—saving information to memory or files, combined with Retrieval-Augmented Generation (RAG). RAG is the classic paradigm of combining external knowledge bases with large models: first using vector retrieval to find relevant document fragments, then injecting them into the prompt for the model to reference. The team also compared GraphRAG with standard RAG. GraphRAG is an advanced approach proposed by Microsoft in 2024 that builds knowledge graphs on top of traditional RAG—first using LLMs to extract entities and relationships from documents to form a graph structure, then dividing the graph into hierarchical communities through community detection algorithms, and finally leveraging the graph structure for multi-hop reasoning during retrieval. GraphRAG significantly outperforms standard RAG in scenarios requiring global understanding and cross-document reasoning, but its construction cost is high. They found that in their scenario, GraphRAG's setup cost was much higher while performing on par with RAG, so they didn't adopt it.

What they adopted was an "LLM Wiki" approach: splitting all content into chunks with cross-links, chunks associated with source data files, where the model only sees a minimal index of about 450 tokens and retrieves layer by layer on demand—this is precisely the idea of "Progressive Disclosure." Progressive Disclosure originates from human-computer interaction design theory, with the core idea of "only showing information when needed." In context engineering, this means not stuffing the entire knowledge base into the prompt at once, but building multi-layer index structures where the model reads the index to decide which topic to dive into, then uses tool calls to get the next layer of detailed content. By analogy, this is like a library's retrieval system—you first check the catalog cards, find the shelf location, then pull out the specific book, rather than hauling every book in the library onto your desk.

Prompt Caching: The Game Changer That Makes Summarization a Trap

This is the most critical insight of the entire presentation. When you ask follow-up questions in a conversation, the system needs to recompute all previous tokens, effectively paying two or three times for the same content. To address this, major providers have introduced Prompt Caching, pre-computing and saving embeddings and KV cache.

To understand why prompt caching is so important, you need to understand Transformer's inference process. When generating each new token, the model needs to compute attention weights between the current token and all preceding tokens, which involves calculating Key and Value matrices—the KV Cache. For a context containing 100K tokens, each time a user asks a follow-up question requires recomputing the KV Cache for all 100K tokens—an enormous computational load. The core idea of prompt caching is: if the prefix of two requests is identical, save the KV Cache from the first computation and reuse it directly for the second. This not only saves GPU computational resources but dramatically reduces Time to First Token (TTFT). Anthropic's Claude, Google's Gemini, and DeepSeek all support this mechanism, though implementation details differ—some require strict prefix matching, others support partial matching.

The key point: these reused cached tokens are extremely cheap. Taking DeepSeek as an example, cached token prices can be as low as 1/50th of the original price. This means when sending ultra-long contexts, the historical portion costs only 1/50th of the price, and only the new user question pays full price.

This creates a fatal contradiction: when you perform summarization or compression, the context becomes "new content," and the cache is immediately invalidated—you must pay full price for these transformed tokens. When you do the math, compression would need to achieve a 50x reduction to actually save money—nearly impossible without quality loss.

Thus the team concluded: summarization may be a trap. Mature frameworks like Claude Code and Codex all use context caching and employ compression methods different from crude summarization. When building your own framework, understanding when to use which technique—and actually testing—becomes crucial.

Test Results: Keeping All Context Wins Across the Board

To validate these techniques, the team built a complete evaluation system. They used Codex to scrape real student Q&As from their website, cleaned them down to 60 pairs; they also constructed multi-turn conversation tasks to test whether models could still recall key facts from early in the conversation after many turns. The evaluation covered 11 preset configurations, including "keep all history," "production default configuration," and six techniques including sliding window, prompt compression, and selective retention.

The results were surprising. In multi-turn conversation tasks using Gemini 3.5 Flash, "don't touch the context, keep all history" won across all three dimensions: memory recall, cost, and speed. Their production default configuration, which they thought was good enough, actually performed worse.

Why was latency also lower? The team analyzed: if you continuously clear tool outputs, the Agent needs to re-retrieve information it already had, which causes more tool calls, ultimately costing more tokens, being slower, and having worse memory recall. This creates a vicious cycle—compression causes information loss, information loss triggers repeated retrieval, repeated retrieval increases latency and cost, and newly retrieved content further bloats the context.

From Gemini to DeepSeek to Local Deployment: Triple Validation of Cost

Early experiments cost nearly $600, prompting the team to switch to cheaper models. They found DeepSeek V4 Flash dramatically reduced costs, and because of the 50x cache discount, "keep everything" remained the optimal solution.

The memory test data was highly compelling: when keeping all context, the model accurately found specific details mentioned by students 95% of the time; once summarized and compressed, accuracy plummeted to 32%. The reason is straightforward—summarization removes necessary details. The team also extended sessions to 800K tokens and found that retrieval of "unique facts" showed almost no context rot, with only fuzzy facts dropping to half performance.

On the cost front, testing confirmed that "the configuration sending the most tokens was actually the cheapest"—because 97% of tokens were cached. For a 1.78 million token long session, keeping everything still led.

But the story shifted at scale and local deployment. When student volume reaches the 100K level, Gemini costs roughly $40,000 per month, while DeepSeek costs only about $1,900. When running locally on a MacBook, limited by the 32K context window, even a single lesson's content won't fit, and caching completely fails—at this point RAG retrieval becomes the only viable approach, with local document retrieval achieving 100% accuracy, but forcefully stuffing the window leads to a catastrophic 340 seconds per token output.

Regarding retrieval strategy, pure semantic search (Dense RAG) performs well at 50K-200K tokens (about 80%), but at 400K, recall drops to zero for facts buried in the middle. This is exactly the "Lost in the Middle" phenomenon manifesting in retrieval scenarios. BM25 keyword retrieval, however, consistently maintains 100%. BM25 is a classic frequency-based retrieval algorithm belonging to sparse retrieval methods, scoring by calculating query term frequency in documents (TF) and rarity across the entire corpus (IDF), extremely sensitive to exact keyword matches. Dense Retrieval uses neural networks to encode text into high-dimensional vectors, measuring semantic similarity through cosine similarity, excelling at handling synonyms and semantically similar but differently worded queries, but potentially missing exact terminology matches. This is precisely why the team ultimately adopted Hybrid Search—typically using methods like Reciprocal Rank Fusion (RRF) to merge result lists from both approaches, proven in practice to be more robust than any single method.

Final Decision and Core Takeaways

Combining all experiments, the Towards AI team ultimately chose: DeepSeek V4 Flash + Hybrid Search as their cloud solution. For memory management, they "keep everything" and only trigger compression after exceeding 30K tokens.

The most important takeaway is: don't compress context by default. You must first clarify your real constraints—whether it's cost, latency, context window limitations, or hardware conditions—then find targeted solutions. Caching mechanisms have completely rewritten the economics of context engineering, making "keep everything" the quality-cost-speed triple-win choice in many scenarios. Only when forced into local deployment or ultra-large scale do RAG and hybrid search return to center stage.

This finding also serves as a warning to the entire AI Agent development community: many techniques considered "best practices" (such as aggressive summarization, sliding windows) may have been formed as heuristics before caching mechanisms became widespread. When infrastructure undergoes fundamental changes, engineering decisions must be re-evaluated accordingly. Avoiding blind optimization and making data-driven decisions may be the most important methodology in context engineering today.