Context Engineering in Practice: Why Keeping Everything Beats Compression and Summarization

It Started with a Frustrated User

Almost everyone who has used an AI Agent has experienced this: you repeatedly explain your needs to the agent, but it keeps doing exactly what you don't want. Eventually, you angrily type out a piece of feedback, hoping it will finally "learn." This real-world context engineering research, shared by the Towards AI team at the AI Engineer conference, starts from exactly that universal sense of frustration.

They argue that the root cause usually isn't the model "getting dumber" — you don't need to rush and switch from Claude to Codex. The real problem is that the context is getting stuffed full, and quality keeps degrading — a phenomenon known as "Context Rot." For a company like Towards AI, which provides courses and an AI Tutor for AI engineers, a single bad interaction can directly lead to a student demanding a refund.

To systematically address this, the team ran extensive experiments on their open-source AI Tutor, publishing all results and code on HuggingFace. This article distills their core findings — including the most counterintuitive conclusion: in many situations, compressing nothing and preserving the full context is actually the optimal strategy.

Two Core Problems in Context Engineering

Context engineering is fundamentally about deciding "what the model can see on each call." This is difficult because of two fundamental constraints in large language models.

The Finite Context Window

A model's context window is limited. System prompts, tool definitions, chat history, old tool outputs, retrieved course snippets, and user questions all compete for the same finite space. The more you cram in, the worse the output quality — and the higher the cost, since you're paying for more tokens.

Through analysis, the team found that the biggest bottleneck for expanding context is "old tool outputs": previously retrieved snippets, all tool call/result pairs, and the agent's past search history. These are not only expensive — they actively degrade answer quality through context rot.

The Stateless Model

Models themselves are stateless. When a student reopens the AI Tutor, if you haven't built any additional mechanism, the model has no memory of anything that happened before. This leads to two areas of work: within-session context management and cross-session memory. This research focuses on the former — because if a single session is already broken, cross-session memory is beside the point.

Worth noting: the team observed an industry trend toward more, smaller skills. The idea is to build precise, small skills, load them on demand one at a time, and even spin up dedicated sub-agents for individual skills. This "progressive disclosure" approach can dramatically reduce context consumption.

The Trade-off Between Compression and Prompt Caching

The intuition behind compression is simple: keep the context as small as possible, retaining only what's needed to answer the question and discarding or offloading the rest. The team identified several cheap techniques that don't even require a large model:

• Auto-truncate unusually long tool outputs: keep the head and tail, mark the middle as "truncated," and let the model re-invoke the tool if needed;
• Sliding window: keep only the last N conversation turns;
• Clear outputs for specific tools.

Going further, you can use a (even local, small) language model for "selective retention," "rolling summaries," or — like Claude Code — generate a global summary and reset when the limit is hit.

Caching Changes the Cost Calculus Entirely

However, one key variable completely upends the cost-benefit analysis of compression: Prompt Caching.

Almost every major API now offers caching. Tokens that have already been sent are pre-computed (saving embeddings and KV cache), making them extremely cheap to reuse — on APIs like DeepSeek, up to 50× cheaper. This means that if you send a long but already-cached context, you only pay full price for the new user question tokens.

The problem is: the moment you compress, summarize, or transform the context in any way, the provider can no longer use the cache — because it's now a "new" context. You then pay full price for all those newly transformed tokens.

This leads to a striking conclusion: for compression to actually be cost-effective, you'd need to compress the context by more than 50× without any quality loss — which is nearly impossible in most real-world scenarios. Summarization may therefore be a trap. This is exactly why mature frameworks like Claude Code and Codex rely heavily on context caching rather than blind summarization.

AI Tutor Experiment Architecture and Evaluation Methodology

To validate which context management techniques actually work, the team built a comprehensive evaluation system. Their AI Tutor is intentionally simple: a single ReAct-style agent built on LangChain, working through tool calls and a reasoning loop, with middleware to customize summarization, tool output cleanup, and similar behaviors.

Two Core Tool Designs

The first is a hybrid retrieval tool: over a corpus of 8 million+ tokens (two years of course content plus open-source documentation from LangChain, LlamaIndex, OpenAI, Claude Code, etc.), it combines semantic search with BM25 keyword search, merges the top 30 results, re-ranks to top 5, and caps output at 100K tokens.

The second lets the agent browse the knowledge base file system using bash commands (inspired by Karpathy's wiki approach, with raw, generated, and wiki folders). Interestingly, experiments showed that adding this tool barely improved recall, but made things ~50% slower — because the real student questions used in testing weren't complex enough to benefit from it.

Evaluation Data and Comparison Configurations

The team defined several key concepts: preset (a specific AI Tutor configuration), task (single-turn Q&A vs. multi-turn session), run (running a preset on a task), and bundle (the result JSON). The evaluation data was not synthetic — for single-turn tasks, they scraped real teacher-student Q&A pairs from the website using Codex, cleaned them, and kept 60 pairs. Multi-turn tasks tested "whether, after several turns of conversation, the tutor can recall a fact stated at the beginning."

They compared 11 presets, including "full history" (untouched context), "production default configuration," and 6 other techniques including sliding window, prompt compression, and selective retention.

Experimental Results: Why Keeping Everything Beats Compression

The first round of experiments using Gemini 3.5 Flash (costing over $500) produced a surprising result: for multi-turn factual recall, "don't touch the context — keep the full history" was the best strategy, even outperforming the team's own production default configuration.

More importantly, keeping full history won on all three dimensions simultaneously: higher recall, lower latency, and lower cost. The reason: if you frequently clear tool outputs, the agent is forced to re-retrieve information it already had, generating more tool calls and thereby inflating both token consumption and latency.

Cheap Model + Cache Hits = Quality and Cost Win

To address high costs, the team switched to DeepSeek V4 Flash and re-ran the experiments. Costs dropped dramatically (with up to 50× cache discounts), and "keep everything" still won. In memory tests:

• With full context preserved, the model accurately recalled specific details mentioned by students 95% of the time;
• With any summarization or compression applied first, the hit rate plummeted to 32% — because summarization strips away exactly those critical details.

In one 36-turn conversation totaling approximately 1.78 million tokens, the "keep everything" configuration — which sent the most tokens — was actually the cheapest, because 97% of those tokens hit the cache. Even in long-context experiments up to 800K tokens, the model's retrieval of unique facts remained stable with almost no observable context rot.

Local Deployment: When Constraints Flip the Optimal Strategy

When the scenario shifts to local models, the conclusion reverses. Constrained by MacBook hardware, the maximum context window is only 32K — and a single course lesson already exceeds that length. Once a conversation no longer fits in the window, caching breaks down. "Keep everything" is no longer viable, and local chat memory scores dropped from 92–95% in the cloud scenario to around 33%. Attempting to force context beyond the window's capacity caused the model to take approximately 340 seconds to output a single token — a complete waste.

However, in the local scenario, retrieval (RAG) performed excellently: accuracy reached 100% when handling pasted long documents, with processing times of 25–65 seconds. On retrieval strategy: dense RAG (pure semantic search) saw recall of facts buried in the middle drop to 0% at 400K tokens, while BM25 maintained 100% consistently — which is exactly why the team adopted hybrid retrieval.

Cost Comparison at Scale

Estimated for 100K to 1M sessions per day, DeepSeek costs approximately $18K to $180K per month. For a scale of 1,000 students, Gemini costs roughly $40K/month, while DeepSeek costs only about $1,900/month, with local deployment reducing costs further (though with throughput limitations).

Final Decision and Key Takeaways

Based on all experiments, Towards AI's final choice was: DeepSeek V4 Flash + hybrid retrieval, with a memory strategy of "keep as much as possible," using a 30K token compression threshold only as a last resort. This achieves balance across quality, cost, and latency.

The single most important insight from this research can be summarized as:

Don't default to compressing context. First clarify your constraints, then look for better alternatives.

In concrete terms:

• Memory retention: Full chat history achieves 95% detail recall; summarized versions only 32%;
• Long-context stability: Retrieving a single fact remains reliable even at 800K tokens, with no significant rot;
• Per-turn cost: The configuration that sends the most tokens is often the cheapest, because caching makes resending context extremely inexpensive;
• Scaled deployment: Switching to cheaper models and local solutions can dramatically reduce costs, but requires weighing context window and throughput constraints.

Compression and summarization look "smart," but in the presence of prompt caching, they often cost more than they save. Context engineering may ultimately be more of an art about knowing when not to do something.