Context Engineering: Why Does More Context Make AI Dumber?

From Prompt Engineering to Context Engineering

If the hottest term when we discussed AI applications in the past was "Prompt Engineering," then Anthropic, in its official article Effective Context Engineering for AI Agents, has introduced a more advanced concept—Context Engineering.

Prompt engineering quickly became a serious discipline after ChatGPT went viral in 2022. Its core idea is to guide large language models toward producing desired outputs by carefully crafting the input text, spawning a whole series of techniques such as zero-shot, few-shot, and chain-of-thought. Early developers even jokingly called it "black magic," because tiny changes in wording could produce vastly different output quality. However, as AI applications evolved from "question-and-answer" chatbots into agents capable of autonomous planning, tool invocation, and multi-step execution, simply optimizing a single prompt was no longer sufficient—the amount of information a model must digest across an entire task chain grows exponentially. This is precisely the historical backdrop against which context engineering steps up to take the baton.

The difference between the two can be summed up in a single sentence: prompt engineering cares about "how to write that one instruction well," while context engineering cares about a much grander question—at each step of the model's reasoning, exactly which information should be placed into its context window.

The scope of "context" here is far broader than a single prompt. System prompts, callable tools, external data, historical messages, retrieved content… all of it counts. In other words, we need to level up from "writing one good sentence" to "curating the entire state of the context." This is a leap from the specific to the holistic, and it is the core skill for building reliable AI agents.

Why Context Is a Finite Resource

The pitch we hear most often lately is: "The window has gotten bigger again—now it can hold a million tokens." But Anthropic reminds us to note a counterintuitive fact: being able to stuff something in and whether you should are two different things.

There is a key concept here called "Context Rot": the more tokens in the window, the worse the model becomes at accurately fishing out a specific piece of information from within it.

This isn't the model "slacking off"—it's determined by the architecture itself. Today's large models are all based on the Transformer, where every token must establish a relationship with every other token. n tokens means on the order of n² relationship pairs. As a result, the model has an inherent "attention budget" that closely resembles human working memory—every additional token you stuff in consumes a bit more attention.

The n² constraint here stems from the core of the Transformer architecture—the Self-Attention mechanism—introduced by Google in the 2017 paper Attention is All You Need, upon which nearly all mainstream large models today (GPT, Claude, Gemini, etc.) are built. Self-attention works by having every token in a sequence "attend to" every other token, computing relevance weights between them. This is precisely its strength: the ability to capture long-range dependencies. But the cost is that compute and memory overhead grow quadratically (O(n²)) with sequence length—processing 1,000 tokens requires roughly 1 million relationship computations, while 10,000 tokens surges to 100 million. This mathematical reality means that even if engineering can push the window to the million-token level, the "attention density" the model allocates to each token gets diluted. This is exactly the technical root of the "context rot" phenomenon.

So the guiding principle of the whole article really boils down to one sentence: spend context like a precious, finite resource. Grasp this premise, and all the principles that follow can be derived quite naturally.

What Good Context Looks Like: Three Core Principles

Since context is a scarce resource, what kind of context counts as "good"? Anthropic offers three practical principles.

The System Prompt Must Find the "Right Altitude"

The first is about positioning the system prompt. Don't hard-code complex, fragile logic into a thick instruction manual and expect the model to follow it point by point; but don't go to the other extreme either, offering nothing but vague slogans.

The goal is to find that just-right "altitude"—using the smallest possible set of high-signal tokens to maximally elicit the behavior you want. The ideal state is: few words, but every one of them earns its keep.

Tool Design Should Be Minimal and Clear

The second is about tool design, where Anthropic offers a highly practical litmus test: if even a human engineer can't clearly say which tool should be called in a given situation, then don't expect the AI to get it right for you.

Ambiguous, bloated tool sets are often a major source of agent errors. Rather than piling up a large number of tools with overlapping functions and blurry boundaries, it's better to trim them down so that each tool has a clear responsibility and they are mutually orthogonal.

Give Only Typical Examples, Not Edge Cases

The third is about few-shot examples. The examples you give the model should focus on just a few typical scenarios—don't pile up all sorts of edge cases in the name of "comprehensive coverage." Because an example is essentially that "a picture is worth a thousand words" image—conveying the clearest behavioral pattern with the fewest samples.

How to Feed in Context: Pre-Retrieval vs. Just-in-Time Retrieval

This is the most valuable section of the entire article: in what manner exactly should data enter the model's context? There are two schools of thought here.

The first is the traditional approach, called "pre-retrieval": before the task even starts, use various means to fetch all potentially useful material in one sweep and stuff it all into the context.

The second is the new approach, called "just-in-time retrieval": normally you leave the model only "clues"—such as a file path or a link—and only when it actually needs the data do you use tools to dynamically pull it in.

The debate between "pre-retrieval" and "just-in-time retrieval" is actually a microcosm of the evolution of Retrieval-Augmented Generation (RAG). Traditional RAG belongs to the classic pre-retrieval school: chunk documents, vectorize them, and store them in a vector database; when a user asks a question, first retrieve relevant fragments and then send them all into the model. This approach performs excellently in knowledge Q&A scenarios, but reveals its limitations when facing complex agent tasks—one-shot retrieval often stuffs in large amounts of irrelevant content, wasting the attention budget and introducing noise. Just-in-time retrieval, by contrast, borrows from the agentic paradigm, turning retrieval itself into a tool the model can actively invoke, letting the model decide "what it needs right now" based on the progress of its reasoning—much like the dynamic, on-demand data-fetching philosophy advocated by Anthropic's own Claude Tool Use and the Model Context Protocol (MCP).

Why is the latter more advanced? Anthropic offers an analogy that hits the nail on the head: this is how human cognition works. We don't memorize an entire library; instead, we rely on external indexes like the file system and bookmarks, looking things up when needed. You only need to remember "which shelf the book is on."

Of course, this isn't absolute. The most effective approach is often a hybrid strategy: fetch and save critical data in advance, and let the agent explore for the rest on its own. And it comes down to the most humble principle of all—

Do the simplest thing that works, and don't pile on fancy tricks from the start.

Long-Task Scenarios: Three Moves Against Context Overflow

The most hardcore scenario is the long-task agent—it may run continuously for hours, executing thousands of steps, and its context will sooner or later blow out the window. For this situation, Anthropic offers three moves.

Move one is Compaction: when the window is nearly full, summarize the preceding content and reopen a clean window with that summary. Keep the decisions and key information, and discard redundant details.

Move two is Structured Note-taking: have the model write notes to persistent storage outside the window, then retrieve them on demand later. This is like equipping the model with an "external notepad."

Move three is the Sub-agents architecture: the main agent handles planning and coordination, backed by a group of sub-agents each working in its own clean window. Each sub-agent returns only a condensed summary of a thousand or two thousand tokens, keeping the messy details isolated outside.

Choosing among them is also simple: for long conversations, prefer compaction; for tasks with clear milestones, use note-taking; for complex research tasks that can be explored in parallel, go with the multi-agent architecture.

The Key Layer That Meshes with "Framework Engineering"

After reading this article, my biggest takeaway is that it meshes tightly with "framework engineering." Every component built around the model actually encodes an assumption about "what the model cannot do," and context engineering is precisely the layer specifically dedicated to managing attention.

This layer has one special property: it won't be fully dissolved as models grow stronger. Because the n² constraint is at the architectural level—windows can keep getting bigger, but the rule that "more leads to chaos" won't disappear.

So the conclusion is: no matter how models evolve, the real skill will always lie in that one sentence—at every step, thoughtfully curate exactly which information gets to enter the model's limited attention.

This article is compiled based on Anthropic's official article Effective Context Engineering for AI Agents, staying as faithful as possible to the original viewpoints.

Key Takeaways