AI Agent Context Management in Practice: A Complete Guide from Death Loops to Intelligent Truncation

In AI Agent development, most people focus their attention on prompt optimization while overlooking a far more critical issue—context management. Sally-Ann Delucia, Head of Product at Arise, recently shared a detailed technical talk about how her team broke free from the "death loop" of context management while building their AI Agent "Alex." These experiences are invaluable for every developer building AI Agents today.

Why Context Engineering Matters More Than Prompt Engineering

In mid-2023, the concept of "Context Engineering" started gaining traction in AI circles. Andrej Karpathy publicly stated on X that he values context engineering over prompt engineering. Karpathy is one of the most influential technical leaders in AI, having served as Tesla's AI Director and as a founding team member of OpenAI. His perspective sparked widespread discussion because it shifted the focus from "how to word instructions" to "how to organize information"—a much deeper system design problem. Sally-Ann strongly agreed, offering a concise definition:

The best context strategy is one that lets your Agent remember what it needs to remember and forget what it doesn't.

Many people think context management is just "stuffing content into the context window," but true context engineering is about strategically selecting what the model can see. Here's the key technical background: An LLM's context window refers to the maximum text length the model can process in a single inference call, typically measured in tokens. Tokens are the smallest units the model uses to process text—one English word corresponds to roughly 1-2 tokens, and one Chinese character typically maps to 1-2 tokens. Early GPT-3.5 had a context window of only 4K tokens, while today's models like Claude and GPT-4o have expanded to 128K or even larger. However, a bigger context window doesn't necessarily mean better results—research shows that when context becomes too long, the model's attention to information in the middle decreases significantly (the "Lost in the Middle" phenomenon).

Therefore, context management isn't simply about staying within token limits—it's about carefully selecting the most important information. If an Agent doesn't have the right context, it gives poor answers; poor answers mean users won't use your product. Context management is fundamentally a product and user experience problem, not just an engineering problem.

The Vicious Cycle: More Failures, Bigger Context

Alex is an AI Agent built on top of Arise's observability platform, handling large volumes of trace and span data. Let me explain these concepts: Trace and Span are core concepts in distributed systems observability, originating from Google's Dapper paper and the OpenTelemetry standard. A Trace represents a complete request chain—the entire process from when a user initiates a request to when a result is returned. A Span is an operational unit within a Trace, such as a database query, an API call, or a model inference. In AI Agent scenarios, every tool call and every LLM inference generates a corresponding Span, and these Spans nest together to form a complete Trace. As an AI observability platform, Arise's core function is collecting and analyzing these Trace/Span data to help developers understand Agent behavior and performance bottlenecks.

The team adopted a "build Alex with Alex" strategy—if Alex could help the team itself develop more efficiently, it could certainly help users too.

However, they quickly fell into a vicious cycle: Alex runs on trace and span data → span data keeps growing → data volume exceeds context limits → Alex fails → the failure itself generates new span data → context inflates further → fails again…

The system was both analyzing data and being constrained by it. The team realized that without solving the context management problem, Alex could never function properly.

Three Steps to Break Free: From Naive Truncation to Intelligent Memory

Step 1: Naive Truncation—Simple but Fragile

The most intuitive approach was truncation: take only the first 100 characters of context and discard the rest. This worked in simple scenarios, but the Agent quickly developed "amnesia." After a user asked their first question and followed up with related details, Alex had no idea what was being discussed—every follow-up felt like a brand new conversation.

Lesson: Over-truncation breaks the reasoning chain, causing the Agent to lose multi-turn conversation capability.

Step 2: Summary Compression—Sounds Perfect, Actually Unreliable

Since LLMs are good at summarizing, why not have them compress all context into summaries? This seems like an obvious solution, but the actual results were highly unstable. The problem: you can't control what the LLM considers "important."

There are deep technical reasons behind this. LLMs have inherent uncertainty when generating summaries, closely related to the model's sampling mechanism. The model samples from a probability distribution when generating each token—even with temperature set to 0 (greedy decoding), different input contexts can cause the model's judgment of "importance" to shift. Additionally, LLMs are prone to "hallucination" during summarization—they may omit critical details or introduce information not present in the original text. In AI Agent scenarios, this uncontrollability is especially dangerous: if a summary loses a critical tool call parameter or intermediate reasoning step, the subsequent reasoning chain completely breaks down.

Lesson: Summaries lack control over "importance" and aren't suitable as a core context management strategy.

Step 3: Intelligent Truncation + Memory Storage—The Actually Viable Solution

The solution that ultimately worked was a combination of intelligent truncation and memory storage:

• Preserve the head: Take the first 100 characters (key context from the conversation's beginning)
• Preserve the tail: Take the last 100 characters (the most recent interaction)
• Truncate the middle: Remove the middle section, but store it in memory
• Retrieve on demand: The Agent can proactively look back and retrieve previous tool call results or conversation content from memory when needed

Notably, this "keep head and tail, truncate middle" strategy aligns perfectly with the "Lost in the Middle" research findings mentioned earlier—the model itself pays less attention to middle information, so truncating the middle has relatively minimal impact on reasoning capability.

The core principle: Context determines what the model currently sees; memory determines what information survives. These two need to work together but remain relatively independent. This solution has been running stably for months with significant results.

Long Sessions: The Silent Killer of AI Agents

Even with intelligent truncation, the team encountered a new problem: users don't proactively restart conversations. In Alex's use case, users switch between different pages while continuously asking questions in the same conversation. Conversation turns grew from fewer than 10 to over 20.

Failures from long sessions often appear late, only discovered when users report issues or the team proactively inspects data. To address this, the team introduced Long Session Evals: load the first 10 turns of conversation, then test performance on turn 11.

Evals are the quality assurance system in AI Agent development, similar to automated testing in traditional software development but fundamentally different. Unlike traditional unit tests, Agent evaluation must handle non-deterministic outputs—the same input may produce different but equally valid outputs. Common evaluation methods include: rule-based exact matching, LLM-as-Judge (using another LLM to judge output quality), and human annotation comparison. The long session evaluation proposed by the Arise team is a specialized evaluation strategy targeting context degradation, detecting performance decline in long conversations by simulating real multi-turn dialogue scenarios.

This way, context degradation becomes a testable, quantifiable metric—no more passively waiting for user feedback. This evaluation method allows the team to discover weak points in context management during the development phase rather than having problems surface in production.

Sub-Agent Architecture: A Divide-and-Conquer Context Strategy

Even with truncation and evaluation, some tasks generate too much data for a single Agent. For example, search tasks—Alex needs to search data in the Arise platform, and a single trace stack might contain hundreds of spans involving multiple queries and extensive intermediate reasoning.

The team's key insight: Not all context needs to exist within the same Agent.

Sub-Agent architecture is a classic pattern in Multi-Agent System design, similar in software engineering to the microservices philosophy—splitting a monolithic application into multiple single-responsibility, independently running services. Currently, mainstream Agent frameworks like LangGraph, CrewAI, and AutoGen all provide native support for multi-Agent orchestration, and this architectural pattern is becoming industry standard practice.

Before the redesign: Main conversation, chat history, heavy data, and search all crammed into a single Agent's context, causing frequent overflow.

After the redesign:

• Main Agent: Retains only conversation context, stays lightweight, plays the "Orchestrator" role
• Sub-Agent: Handles data-intensive operations with independent context space, plays the "Specialist" role
• Collaboration mechanism: The main Agent delegates heavy tasks to sub-Agents, which process them and pass results back
• Shared memory: Both can retrieve needed context from memory storage

This architecture became a true "game changer." The team subsequently launched multiple sub-Agents, handling all data-intensive operations with this pattern, effectively avoiding single-Agent context overload. The advantages of this architecture go beyond solving context inflation—it also brings better maintainability and scalability, as each sub-Agent can be independently optimized, tested, and iterated.

Frontier Challenges Still Being Explored

Despite significant progress, Sally-Ann acknowledged the team still faces multiple challenges:

1. Extremely large context still causes crashes. Since Alex is an Agent running on Agent data, customer system prompts, user messages, and conversation histories are themselves the objects of analysis. As customer data grows, context problems amplify. The current response pattern is still to continue splitting into sub-Agents.

2. Long-term memory hasn't been achieved yet. The current memory mechanism is essentially "context with storage," not true long-term memory. Users want to reference previously discussed issues and even continue analyses across new conversations. True long-term memory requires solving multiple technical challenges: how to persist key information across sessions, how to build efficient memory indexing and retrieval mechanisms, and how to handle memory staleness and conflicts. The team is developing true long-term memory capabilities.

3. Context selection still relies on heuristic rules. The "first 100, last 100" strategy works, but the team hasn't established principled context budgets or context quality metrics. They currently rely primarily on evals to determine whether context selection is correct. Ideally, the system should dynamically adjust context allocation strategies based on task type and complexity, rather than using fixed truncation parameters.

Interestingly, when Claude Code's code was made public, the team discovered that Anthropic was using similar truncation and compression strategies. Claude Code is Anthropic's AI programming assistant that can autonomously complete code writing, debugging, and refactoring in terminal environments. The public release of its implementation code revealed an important fact: even the most cutting-edge AI labs haven't found a fundamental breakthrough for the engineering challenge of context management. Current context management relies more on engineering experience and heuristic rules than theoretically optimal solutions, which also means there's enormous room for innovation in this space. The Arise team had hoped to find some "secret weapon" from it, only to discover that everyone faces remarkably similar challenges and solutions.

Key Takeaways

From nearly a year of the Arise team's practical experience with AI Agent context management, we can distill several key insights:

1. Context management is an iterative process—there's no one-and-done solution
2. Context engineering, memory management, and evaluation systems are all indispensable
3. The root cause of Agent failures is often context, not prompts
4. Sub-Agent architecture is an effective paradigm for data-intensive tasks
5. Long session evaluation is the key method for detecting context degradation

As AI Agent development matures, context management is evolving from a "side issue" to a "core engineering challenge." Whoever can better solve "making Agents remember what they should and forget what they shouldn't" will build truly reliable AI Agent products.