[KongchangAI]
· 3 min read· 1,523 words

ReAct Runtime Context Trimming: Why Claude Code Acts at 70%

ReAct Runtime Context Trimming: Why Claude Code Acts at 70%

Context trimming keeps agents reliable via grouped discards, on-disk indexing, and preserving decision paths.

The article argues that context is a finite resource with diminishing marginal returns — the more information, the lower the model's recall precision — making trimming the linchpin of agent reliability. Claude Code starts compressing at around 70% usage rather than waiting for the ceiling. Tool calls dominate context usage and must be discarded in groups with their results to avoid orphan messages. It also explains that codebases can be retrieved directly without RAG thanks to natural relationship chains like imports, that trimmed info is persisted to disk and indexed for traceability, and that system prompts, text reasoning, and task decision paths must never be dropped.

Why Context Trimming Is the Linchpin of Agent Reliability

When building AI Agents, many people focus on tool calls and prompts while overlooking the one thing that truly determines whether an agent is reliable: context management. In Episode 17 of Learn AI Agent Development Without Spending a Dime on Bilibili, creator Xiao Fu Ge makes this point sharply — context is a finite resource with diminishing marginal returns. Even if a model supports an extremely long context, cramming all information in at once means the more tokens there are, the lower the model's recall precision becomes; attention gets diluted, and it becomes harder for the model to figure out what you actually want.

So the question isn't "can it all fit" but "what should go in." This is also a key constraint in Claude Code's design: it doesn't wait until context hits the ceiling, but rather begins compressing and trimming at around 70% usage, reassembling the most precise, most necessary information before sending it to the LLM.

The telescope metaphor for context compression

The Telescope Metaphor: Limited Field of View, See It as You Turn

Xiao Fu Ge uses a vivid metaphor to explain the trimming process: a person's field of view is limited — you can't take in everything at a glance, so you turn your head and look piece by piece. Agent runtime works the same way: the system prompt, tool definitions, user messages, assistant messages, and tool results keep sliding forward in chronological order, and you must trim progressively as they slide so the conversation can continue to advance correctly.

Context Window and Attention Dilution: An LLM's context window refers to the maximum number of tokens it can accept in a single inference. Mainstream models have expanded from early 4K and 8K to 128K and even millions. But a bigger window doesn't mean effective utilization — Transformer self-attention computes association weights between every token and every other token in the sequence, and the longer the sequence, the more dispersed the attention each individual token receives, making key instructions easy to drown out. The industry calls this the "lost in the middle" phenomenon: information located in the middle of the context has noticeably lower recall than information at the beginning and end. This explains why cramming in all history actually makes agent performance worse, and it's the fundamental reason trimming strategies exist.

Who's Eating Your Context

Before understanding trimming strategies, you need to know who is occupying the context. Xiao Fu Ge breaks down the sources clearly:

  • System prompt: Describes what the agent can do, which tools it has, how it works, and what constraints exist — essentially fixed.
  • Skills: Used in layers — the first layer only holds tool descriptions, and actual content is extracted only when needed; also fixed.
  • User messages: Usually just one, describing the task goal.
  • Assistant messages and tool calls: The model executes step by step once it starts polling — this is the real bulk.

Tool calls almost always take up the most. Tools need to retrieve large amounts of information, pass parameters, execute, get results back, and then analyze — especially in scenarios involving an embedded browser dynamically clicking buttons to scrape data, where context volume balloons rapidly. So when trimming, tool-related messages are often the part that gets cut the most.

Retrieving code relationship chains through tools

Why Codebases Don't Need RAG

Here's an easily overlooked engineering insight. Early on, the Claude-related team generated a knowledge base for the entire project and then did vector-matching retrieval, but found it wasn't accurate. Later, mainstream Agent tools switched to retrieving the project directly as a file repository — the reason is simple: when you write code, you inevitably have imports, and the call relationships between classes and methods are already explicitly marked, whether it's Python, Java, or C#. This natural relationship chain makes progressively unlocking code accurate and reliable.

By contrast, content like legal statutes and company policies has no relationship chain and can't describe a path from A to B — that's why it needs a RAG knowledge base or knowledge graph to build an index. This is why many code Agents don't re-run RAG over the entire knowledge base at all — direct retrieval is simply the most accurate approach.

The Applicable Boundary of RAG and Vector Retrieval: A typical RAG (Retrieval-Augmented Generation) pipeline chunks documents, converts them into vectors using an embedding model, stores them in a vector database, and recalls the top-k fragments by semantic similarity at query time. Its strength lies in handling unstructured text lacking explicit associations, but similarity matching is inherently fuzzy — it easily recalls fragments that are semantically similar but off-topic, and it loses the structural relationships between documents. Code is the opposite: import statements, class inheritance, and function calls form an explicit directed graph, and an Agent can follow symbol references to jump precisely, achieving "look it up and be certain." Therefore code Agents are closer to an IDE's symbol index (like LSP) than to semantic retrieval.

The Basic Unit of Trimming: Discard Group by Group

Trimming isn't about deleting messages at random. Tool calls and tool results are paired and must be discarded as a group — dropping only the call without the result, or keeping only the result without the call, produces "orphan messages" that make the conversation unable to continue. Xiao Fu Ge demonstrates this with the classic case of checking the weather in Hangzhou and then computing: the assistant message triggers a tool call, the tool returns a result, and this group is either kept in full or dropped in full.

Trimming must handle historical messages together

Where Do Discarded Messages Go

One design detail of Claude Code is worth noting: trimmed content doesn't vanish into thin air — it gets written to local files. This is why you'll notice disk usage growing quickly when using various Agent tools — todos, indexes of the entire process, and context logs all get persisted to disk. Unless you clean them up manually, they keep accumulating.

This design addresses a real need: certain scenarios (like chess or Go game review) require tracing back a long chain to see how things happened step by step. Writing information to local storage and building an index allows it to be retrieved later. The standard approach is: as context advances, trim the earliest information, build an index, advance further, trim again — repeating in a loop.

Persisted information keeps accumulating unless manually cleaned up

Orphan Messages and the Message Pairing Constraint: Most LLM APIs require tool call messages (tool_calls) to appear in strict pairs with their result messages (tool role), with matching order and IDs. If trimming deletes only one side, the request will be directly rejected by the API or cause abnormal model behavior — this is the technical root of the "orphan message" problem. Therefore trimming algorithms typically operate with a "turn" or "call group" as the minimum unit, maintaining an indivisible message group boundary. A common engineering practice is to tag each message group with a trimmable flag and a reference count, ensuring atomic removal of the entire group when deleting.

Which Messages Must Never Be Dropped

In the benchmark version given by Xiao Fu Ge, trimming priorities for different messages are clearly distinguished:

  • System prompt (behavior contract): Assembled separately at request time, naturally immune to trimming — trim the system and nothing runs.
  • User messages: Task constraints, usually discardable after early processing, but recommended to keep a fallback for future retrieval.
  • Text reasoning: Prioritized for retention — tests show reasoning content is valuable, so try not to drop it.
  • Tool pairs: Discarded as a whole group, never producing orphan messages.

Comparing the teaching example with the production version, the core principle is consistent: never drop the task's decisions and paths. Either keep the record, or write a list, or write it into a todo file — ensure state is traceable.

Engineering Complexity Seen Through the Teaching Example

The sample code in the course implements a simplified trimming loop: assemble the system prompt and historical messages, call the tool, check whether the tool has a return, fill in historical messages, handle normal completion and exceptions, and finally return the result. This is just a basic version with no indexing — a real Agent project also needs to build an index to support later retrieval.

Message assembly in the conversation loop

Production-grade tools like Claude Code and Codex are far more complex: recording historical messages, trimming after collection completes, and retaining core context. Xiao Fu Ge's advice is that after finishing the simplified version, go read Claude engineers' public articles on context rot and compression tools to understand the full chain of compression, trimming, clearing, and memory management — these materials are provided along with the course code.

To sum up this section in one sentence: if something can be said in one sentence, don't say it in a hundred. The essence of context trimming is to ensure the agent always knows clearly, at runtime, what the user actually wants.

#ReAct#context trimming#Claude Code#AI Agent#RAG#context window#tool calling
Share:

Related articles