Context Engineering: A Systematic Approach to Stop AI Agents from 'Talking Nonsense'

Why Models 'Talk Nonsense'

Imagine a whiteboard already covered in writing. What happens if you keep forcing new content onto it? The key points get squeezed into the corners, critical information gets overwritten, and the reader can't grasp the main ideas at all. The context window of a Large Language Model (LLM) is exactly like this whiteboard—conversation history, tool call results, and reference documents all pile up in a limited space. When the window is stuffed full, the truly important information gets crowded out, and the model starts to 'talk nonsense.'

The Context Window is the maximum number of tokens an LLM can 'see' in a single inference pass. A token is not simply equivalent to a character—typically an English word is about 1-2 tokens, and a Chinese character is about 1-2 tokens. The early GPT-3 had a context window of only 4,096 tokens, while modern models like GPT-4 Turbo and Claude 3 have expanded to 128K or even millions of tokens. However, a larger window doesn't mean the problem disappears. Stanford University's 2023 research paper Lost in the Middle experimentally confirmed that when relevant information is placed in the middle of a long context, the model's accuracy drops significantly, while information at the beginning and end is prioritized for encoding.

This phenomenon stems from how the attention mechanism works in the Transformer architecture—the distribution of positional encodings and attention weights is not uniform, and the model is naturally more sensitive to information at the sequence boundaries. It's worth noting that positional encoding itself has been continuously evolving: early approaches used fixed sinusoidal-cosine encoding, while modern long-context models widely adopt relative positional encoding schemes such as RoPE (Rotary Position Embedding) or ALiBi to better extrapolate to ultra-long sequences never seen during training. RoPE encodes positional information by applying a rotational transformation to the query and key vectors in complex space, naturally incorporating relative distance into the attention score computation, which gives the model better extrapolation when handling ultra-long sequences. ALiBi takes a more aggressive linear penalty approach, directly adding a negative bias proportional to positional distance onto the attention scores, requiring no learnable positional parameters, and demonstrates particularly strong generalization to sequences beyond the training length during inference. Even so, the 'boundary preference' of attention weights remains prevalent across various architectures, becoming a structural constraint that must be confronted in long-context engineering. This also explains why, even with an ultra-long window, careful design of information placement order is still needed in practical engineering: the most critical instructions and reference materials should be placed at the beginning of the prompt, while summarizing constraints go at the end, to maximize the model's effective attention. This characteristic makes managing the arrangement order and information density of context especially important.

Often, we mistakenly believe the model is 'dumb,' but the truth is usually: it's disconnected from your world. The model doesn't know your private documents, nor what happened yesterday. When it lacks sufficient context, it tends to confidently fabricate answers—this is what's known as 'hallucination.'

An LLM's 'hallucination' is not a random error but has deep statistical mechanistic roots. An LLM is essentially a probabilistic predictor that, during training, learns 'what the next word is most likely to be' from massive amounts of text. When the model lacks relevant context, it will still generate plausible-sounding content based on statistical patterns in the training data—even if that content is factually incorrect. Hallucinations can be divided into two categories: first, 'intrinsic hallucination,' where the generated content contradicts the provided reference materials; second, 'extrinsic hallucination,' where the generated content cannot be verified by any material. Understood from an information theory perspective, hallucination is essentially the model performing 'maximum likelihood extrapolation' in knowledge gaps—the model tends to generate word sequences that co-occur frequently in the training corpus, rather than admitting the boundaries of its own knowledge. This mechanism causes the model's hallucinations to be extremely superficially convincing: fluent prose, standardized formatting, yet completely distorted at the factual level, making them harder to detect and correct than obvious errors. Research shows that providing high-quality, precise contextual information is currently one of the most effective engineering means to reduce hallucination, which is also the core driving force behind the rise of RAG (Retrieval-Augmented Generation) technology.

RAG was formally proposed by Meta AI in the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Its core idea is to decouple parametric knowledge (stored in model weights) from non-parametric knowledge (external document libraries): model weights are responsible for reasoning capability, while external knowledge bases are responsible for factual accuracy. This architecture reduces the cost of knowledge updates from expensive full fine-tuning to incremental updates of the document library, while providing traceable citation sources, greatly alleviating the hallucination problem, and has become the standard paradigm for enterprise-grade AI applications. From a technical evolution perspective, RAG has developed from early Naive RAG (single-round retrieval) into Advanced RAG and Modular RAG. Advanced RAG introduces pre-retrieval optimization (such as query enhancement) and post-retrieval optimization (such as reranking and context compression). Modular RAG further decouples the retrieval components to support flexible composition, and introduces Iterative RAG and adaptive retrieval mechanisms, enabling the system to dynamically determine retrieval strategies based on question complexity, representing the cutting edge of current RAG engineering.

Context Engineering is the systematic method for organizing this whiteboard. It is not a single point technique, but a complete component system that determines how information enters, how it is organized, and how it is efficiently utilized by the model.

Core Components of Context Engineering

Context engineering is a system composed of multiple collaborating components, which can be roughly divided into three layers of responsibility: the Agent handles decision-making, query enhancement and retrieval handle finding information, and prompts, memory, and tools handle organizing action.

Agent: The System's Traffic Dispatcher

The Agent plays the role of a traffic dispatcher within the entire system. It decides which direction information flows, and it also decides which tool to call and when. During the execution of complex tasks, the Agent must also continuously maintain task state—recording what has been completed and where the current progress stands.

More critically is its fault tolerance: when a certain execution route fails, the Agent must be able to re-plan the path, rather than getting stuck or giving up outright. This dynamic dispatching and self-repair capability is the essential characteristic that distinguishes an Agent from a simple call chain. Modern Agent architectures are typically based on the ReAct (Reasoning + Acting) framework, proposed by Google DeepMind in 2022 and published at ICLR 2023. Its core idea is to alternate the reasoning process (Thought) with external tool calls (Action), updating the reasoning state through feedback (Observation) after each action, forming a Think→Act→Observe iterative loop. The key breakthrough of this design is that the reasoning process itself is explicitly written into the context, enabling the model to maintain coherent decision logic in complex multi-step tasks. Compared to early Agent schemes that relied only on single-step prompts, ReAct's explicit reasoning chain greatly reduces the probability of the model 'forgetting the goal' in long-horizon tasks, and makes debugging and interpretability possible—developers can directly review the model's reasoning trace to pinpoint the specific step where decision errors occurred.

In practical engineering, the challenge with ReAct is that the reasoning chain continuously consumes context tokens—an Agent executing 20 steps of operations might consume tens of thousands of tokens in its historical reasoning chain. To address this, engineers typically adopt strategies such as sliding windows (retaining only the last N steps of reasoning history) or hierarchical summarization (compressing early reasoning chains into concise summaries). The Plan-and-Execute architecture, as an evolution direction of ReAct, separates the planning phase from the execution phase to reduce the context burden of a single inference pass, making it suitable for long-horizon tasks with many steps. How to compress historical reasoning traces and retain only information useful for the current decision has always been a key engineering problem in Agent development.

Query Enhancement: Translating Colloquial Questions Clearly

Questions raised by users are often messy and colloquial. Query Enhancement is like a translation counter that first rewrites vague colloquial questions into clear, standardized expressions, then expands them—unifying terminology and supplementing keywords.

When facing complex questions, it will further break them down, decomposing one big question into multiple independently processable sub-queries. This way, the subsequent retrieval stage can precisely hit relevant information, rather than being led astray by a vague question. Common query enhancement techniques also include HyDE (Hypothetical Document Embeddings)—proposed by Carnegie Mellon University, whose core insight comes from the asymmetry of semantic space: a user's original question is usually a brief interrogative sentence, while relevant documents are declarative long text, and the distance between the two in vector space is often much greater than the distance between two related documents. HyDE first has the model generate a hypothetical answer document, then uses the vector of that answer to retrieve real documents, leveraging the geometric property that 'an answer is semantically closer to an answer,' significantly improving the recall quality of knowledge-intensive queries. Another important query enhancement technique is Multi-Query Retrieval: for the same user question, an LLM automatically generates 3-5 semantically equivalent but differently phrased variant queries, retrieving each separately and taking the union. The value of this strategy lies in hedging against the 'directional bias' of a single query in semantic space—when the wording of the user's original question happens to have semantic drift from the key expressions in the documents, multi-query can cover relevant content from different angles, significantly improving recall. It is a simple, easy-to-implement engineering practice with stable results.

Retrieval: Like Finding a Book in a Library

The essence of the Retrieval stage is like finding a book in a library. This process has several key steps:

First, the documents must be cleaned up, removing noise and irrelevant formatting; then the documents are split into appropriately sized small chunks (Chunking)—cutting them too large wastes context space, while cutting them too small loses semantic integrity; next, a vector database is responsible for quickly finding the fragments most relevant to the query.

The Vector Database is the core infrastructure of modern RAG systems. Its working principle is: text fragments are converted into high-dimensional vectors (usually arrays of 768- or 1536-dimensional floating-point numbers) through an Embedding Model, and the distance between these vectors in geometric space represents the degree of semantic similarity. During retrieval, the user query is also converted into a vector, and the system quickly finds the semantically closest document fragments through Approximate Nearest Neighbor (ANN) algorithms. Among these, HNSW (Hierarchical Navigable Small World) is the most widely adopted ANN algorithm in industry. Its principle is to build a multi-layer graph structure—sparse at the top, dense at the bottom—where retrieval quickly locates the approximate region from the top layer and then fine-searches at the bottom layer, balancing O(log n) query speed with high recall. Meta's open-source FAISS offers multiple index structure choices, suitable for large-scale offline batch processing scenarios. In addition, ScaNN (Google Research) further balances retrieval precision and memory footprint through anisotropic quantization, representing another evolution direction of ANN algorithms.

In terms of chunking strategies, beyond fixed-size chunking, Semantic Chunking determines split positions by detecting sharp drops in similarity between adjacent sentence embedding vectors, preserving more complete semantic units. Parent-Child Chunking adopts the strategy of retrieving small chunks while inputting large chunks, balancing retrieval precision and context completeness. The choice of chunk size is essentially a precision-recall trade-off: smaller chunks (e.g., 128 tokens) improve the semantic purity of vector representations, making retrieval more precise, but have lower context information density, potentially lacking necessary background during answer generation; larger chunks (e.g., 512 tokens) preserve more complete semantic coherence, but the vector representation gets diluted by multiple topics, reducing retrieval precision. Mainstream vector database products include Pinecone, Weaviate, Chroma, and pgvector. It's worth noting that vector retrieval excels at semantic similarity matching but is sometimes inferior to traditional full-text retrieval (BM25) for exact keyword matching—BM25, based on the term frequency-inverse document frequency principle, has a natural advantage for queries containing exact terms, product model numbers, and proper nouns. Therefore, production-grade systems typically adopt hybrid retrieval strategies, merging the two result sets through the Reciprocal Rank Fusion (RRF) algorithm before reranking. RRF, with its characteristics of requiring no parameter tuning and being naturally robust to the scoring scales of different retrieval systems, has become the de facto standard fusion method in hybrid retrieval.

It's worth emphasizing that the retrieved results cannot be used directly; they still need to go through Reranking, placing the truly most relevant content first before putting it into the context. The Cross-Encoder used in reranking differs fundamentally in architecture from the Bi-Encoder used in vector retrieval: the Bi-Encoder encodes the query and document independently, offering extremely fast retrieval speed but limited precision; the Cross-Encoder concatenates the query and candidate document and jointly inputs them into the model, capturing fine-grained semantic interaction between the two through full attention, outputting more precise relevance scores. In practice, vector retrieval is typically used first to recall Top-50 to Top-100 candidate fragments, then a rerank model refines the selection to Top-5 to Top-10 to put into the context. The core value of this two-stage 'coarse recall + fine ranking' architecture is that a small amount of fine-ranking computation is traded for a substantial improvement in overall retrieval quality, and it has become standard industry practice. This 'clean—chunk—retrieve—rerank' workflow directly determines whether the model can obtain high-quality reference information.

Prompts and Memory: The Foundation for Organizing Action

Prompt: A Clear Work Order

Prompt techniques are like handing the model a work order, on which is clearly written what the task is, what the constraints are, and what the expected output format is.

If Few-shot Examples are needed, these examples must be close to your actual business scenario—generic examples offer limited help. The effectiveness of few-shot learning stems from the 'In-Context Learning' (ICL) capability that LLMs acquire during pre-training—the model can recognize task patterns from the few examples provided in the prompt and generalize them to new inputs, without updating any model weights. Regarding the mechanism of ICL, the mainstream hypothesis holds that it is essentially a form of implicit gradient descent: during the forward pass, the model performs 'soft weight updates' on the examples through the attention mechanism. This capability only emerges significantly after the parameter count exceeds about 10 billion, and is closely related to the 'Emergent Abilities' phenomenon in large models.

In engineering applications, dynamic example selection methods such as KATE (K-nearest Neighbor Augmented in-context Tuning Examples) retrieve examples semantically closest to the current input, delivering a 5-15% performance improvement compared to randomly fixed examples. The example most similar to the target should be placed immediately adjacent to the task description (leveraging the recency effect), fully utilizing the model's attention preference for the end of the sequence. For format-sensitive tasks (such as structured data extraction, code generation), the value of few-shot examples often exceeds detailed textual descriptions. At the same time, the guidance of the reasoning process should also be streamlined—although Chain-of-Thought (CoT) prompting can significantly improve performance on complex reasoning tasks, a detailed reasoning chain occupies a large number of context tokens. In the evolution path of CoT, Zero-shot CoT (triggering reasoning merely with 'Let's think step by step') reveals the activatability of large models' reasoning capabilities; while Tree-of-Thoughts (ToT) further extends the linear reasoning chain into a tree-shaped search structure, allowing the model to evaluate and backtrack among multiple reasoning branches, performing particularly well in planning tasks requiring systematic exploration of the solution space, but at the cost of higher token consumption and latency, requiring trade-offs based on task complexity. Because the prompt itself also occupies context space, if the prompt is written long and verbose, it will instead fill up the whiteboard, which is not worth it.

Memory: The Desk and the Filing Cabinet

The memory system can be analogized to a combination of a desk and a filing cabinet, divided into three levels:

• Short-term memory: Like the files being processed on the desk, it stores immediate information related to the current task, stored directly in the model's context window, disappearing when the session ends;
• Long-term memory: Like a filing cabinet, it preserves the user's factual information and preference settings, can be reused across sessions, and is typically persistently stored through an external database, divided into structured storage (SQL databases) and vector storage;
• Working memory: Preserves the intermediate state of multi-step tasks, ensuring complex workflows don't 'lose the thread.' Technically, this is usually implemented through a Scratchpad mechanism, allowing the model to write intermediate reasoning results into a readable and writable temporary storage area.

Worth noting is that the 'forgetting mechanism' of memory is equally important—drawing on the idea of operating system virtual memory management, dynamic strategies decide which memories stay in 'main memory' (context) and which are archived to 'external storage' (persistent storage). In practical engineering, memory writing usually requires setting explicit trigger conditions rather than passively receiving all information: for example, triggering a write operation only when the user explicitly expresses a preference, completes a key task milestone, or the conversation undergoes a substantial topic shift. Research projects such as MemGPT transplant the paging memory management idea of operating systems into LLM memory systems, by explicitly defining the swap-in/swap-out mechanism between the 'main context' (corresponding to memory) and 'external storage' (corresponding to disk), granting the model the ability to autonomously manage memory, representing a cutting-edge exploration direction for memory system architecture. The key principle is: there must be a threshold for what is worth storing. Storing all information without filtering will only make the memory system bloated and inefficient, in turn dragging down subsequent retrieval and utilization.

Tools: Giving the Model the Ability to Act

Tools give the model the true ability to interact with the external world. But we must soberly recognize: giving a tool does not mean the model can use it well.

A model using a tool is a complete closed-loop process: it must first discover the tool (know which tools are available), then select the tool (judge which one to use for the current task), then fill in the parameters (correctly construct the call input), and after the result returns, it must also observe and reflect—judging whether this call achieved the expected outcome and whether the strategy needs adjustment.

The quality of tool registration and description is equally crucial—the clearer the tool description and the more defined the boundaries, the higher the model's accuracy in the 'discovery' and 'selection' stages; vague or overlapping tool descriptions are often one of the main root causes of Agent call failures. This whole cycle of discovery, selection, execution, and reflection is the key to truly implementing tool capabilities. In the engineering practice of tool descriptions, each tool's specification should include four core elements: the precise boundaries of the function (what it can do, what it cannot do), the types and constraints of parameters (valid value ranges, required vs. optional), typical call examples (covering normal paths and edge cases), and distinguishing notes from similar tools (avoiding the model's confusion in choosing between functionally overlapping tools). Research shows that even for models of equal capability, an Agent equipped with high-quality tool descriptions can have a 20-40% higher success rate in complex tasks compared to schemes with vague descriptions, and the engineering investment in tool descriptions is often severely underestimated.

Four Core Actions to Manage the Context Window

If we were to summarize the operational principles of context engineering in the most concise way, we can remember these four actions:

1. Write: Write important information to external storage, rather than piling it all into the context window;
2. Select: Select only the information the current task truly needs to enter the context;
3. Compress: Compress lengthy conversation history into concise summaries. Compression techniques currently have three main approaches—summarization compression, selective token pruning (such as LLMLingua developed by Microsoft Research Asia, which evaluates the information entropy of each token through a small language model: a low-perplexity token means it can be predicted with high confidence, i.e., high information redundancy, and can be safely deleted; research shows that model performance can remain basically stable while compressing 80% of the content; the subsequent version LongLLMLingua further optimizes for long-context scenarios, able to identify and prioritize preserving key passages highly relevant to the query), and KV Cache reuse at the underlying inference architecture level—for repeatedly occurring system prompts or fixed documents, their attention key-value pairs are pre-computed and cached, avoiding repeated computation with each inference; Prefix Caching extends this optimization to cross-request reuse, reducing time-to-first-token (TTFT) latency and computation cost by 30-50% for high-concurrency scenarios with identical system prompts, substantially expanding usable context capacity while reducing computation cost. Understanding the essence of compression from an information theory perspective: effective context compression is not simple length reduction, but maximizing information entropy density within a limited token budget. This means preserving high-information key facts, decision nodes, and constraints, while deleting highly predictable redundant expressions, transitional phrasing, and background information already implied in the context.
4. Isolate: Isolate noise and potential risk information to avoid polluting the core context. This is especially important when guarding against Prompt Injection attacks—malicious content mixed into the context can hijack the Agent's behavior. Prompt injection was first systematically documented by security researcher Riley Goodside in 2022 and has since rapidly become a core topic in the AI security field, with OWASP listing it as the top security risk among the LLM Application Top 10. Indirect Prompt Injection is particularly dangerous because the attack vector comes from external data processed by the model (web pages, PDFs, emails, database records) rather than the user input itself, making traditional input filtering methods difficult to defend against, and the attack intent completely opaque to end users. Current mitigation strategies include: Instruction Hierarchy training—explicitly teaching the model during fine-tuning to distinguish trusted system instructions from untrusted external data; the principle of least privilege—the Agent gets only the minimum tool permissions needed to complete the task; and Human-in-the-Loop review—forcibly introducing a human confirmation node before executing irreversible operations.

These four actions run through every stage of context engineering and are the golden rule for managing the 'whiteboard space.'

Conclusion: A Clean Whiteboard Makes a Smart Model

The core insight of context engineering is that a model's capability ceiling is, to a large extent, determined by how we organize information for it. It is not merely a Prompt tuning technique, but a complete systematic engineering discipline covering decision dispatching, query enhancement, information retrieval, prompt design, memory management, and tool invocation.

For developers building AI Agents, rather than complaining that the model is 'not smart enough,' first examine: have you given it a clean, focused, information-rich whiteboard? Doing the four things—Write, Select, Compress, Isolate—well often brings more substantial effectiveness improvements than switching to a larger model.

Key Takeaways