How to Save Tokens on LLM Retries? Optimization Strategies for Large-Context Agent Workflows

When building production-grade AI Agent workflows, a frequently overlooked yet costly problem surfaces: every time an LLM's output fails validation and needs to be retried, sending the full context again causes token consumption to balloon rapidly. An engineer shared this real-world challenge on Reddit, proposing several possible solutions and sparking an in-depth discussion on Agent context management.

The Problem: The Retry Black Hole After Validation Failure

The developer was building an internal AI workflow for processing XMP files — containing source tables, columns, and metadata. The overall pipeline looked roughly like this:

XMP Metadata → LLM → MCP Tool → Dataflow → Validation

The pain point was concentrated in the failure-handling stage. When the initially generated dataflow failed validation, the current approach was to resend the entire original text context along with the error message, asking the LLM to correct it. In other words, the retry request looked like:

Original XMP + Instructions + Generated Dataflow + MCP Error → LLM Retry

When the XMP metadata is large, this "full resend" strategy causes token consumption to multiply. A validation failure might involve a small error in a single node, yet you pay for the entire context all over again — neither economical nor elegant in a production environment.

The Core Tension: Large Input vs. Small Fix

At its heart, this is a structural tension that exists broadly across Agent workflows: the input context is large, but retries usually only need a small portion of it.

The generation phase genuinely requires complete metadata to understand table structures and field relationships, but the repair phase typically targets just one failing transformation node. Treating the entire context as an "atomic unit" to be passed repeatedly means paying the full cost every time you make a minor tweak. Identifying and breaking this coupling is the key optimization lever.

Six Candidate Optimization Approaches

The original poster outlined several directions he was considering, each reflecting different architectural tradeoffs:

1. Externalize Context + Retrieve on Demand

Keep the original metadata outside the conversation (e.g., in external storage or a vector database), and retrieve only the relevant portion during retries. This essentially applies the RAG philosophy to the retry loop, preventing large static text from occupying the conversation window.

RAG (Retrieval-Augmented Generation) is an architectural pattern that combines external knowledge bases with LLMs: at inference time, relevant fragments are dynamically retrieved based on the current query, rather than stuffing all knowledge into the context upfront. Applying this idea to retry scenarios means the original XMP metadata or table schema documents are chunked, vectorized, and stored in a vector database (such as Pinecone, Weaviate, or pgvector). A retry request then includes only the fragments relevant to the failing node, rather than the full metadata. The tradeoff is maintaining a retrieval pipeline, where retrieval quality directly determines repair effectiveness — if the relevant context isn't recalled, the LLM may generate an incorrect fix due to insufficient information. Therefore, retrieval strategies (such as similarity thresholds and top-k recall counts) need to be tuned in alignment with the granularity of validation errors.

2. Send Only the Failing Node and Validation Error

Instead of sending back the entire generated dataflow, precisely pinpoint the failing node or transformation and send it along with the validation error to the LLM. This is the most straightforward "subtraction" approach, provided the error can be accurately localized.

3. Maintain Structured Agent State

Manage the workflow using a structured state object rather than simply appending to a conversation history. Structured state lets you selectively assemble the fields needed for each request, rather than linearly stacking historical records.

The concept of structured Agent state draws from the State Machine design paradigm. Unlike the linear appending of ordinary conversation history, a state machine explicitly models key process variables — such as the current node, the list of validated transformations, and the set of failing nodes awaiting repair — as queryable, updatable fields. Mainstream Agent frameworks like LangGraph and LlamaIndex Workflows provide built-in state management abstractions, allowing developers to define input/output schemas for each node. In retry scenarios, the framework can automatically trim the request context based on the failed_nodes field in the state, passing only the necessary fields rather than copying the entire messages list to the next LLM call. This approach also naturally supports checkpoint resumption and observability, making it recommended infrastructure for production-grade Agent workflows.

4. Summarize and Compress Previous Attempts

Before retrying, summarize or compact the previous attempt — preserving key decision points while discarding redundant details. This is a common technique in long-conversation Agents, but be aware that compression may lose critical constraints.

5. Have MCP Tools Return Compact Structured Errors

Refactor MCP tools to return a small, structured error object rather than regurgitating the full dataflow or context. The output design at the tool layer directly impacts token efficiency in the Agent loop — a point that is consistently underestimated in practice.

6. Generate Once + Targeted Repair

Perform the full dataflow generation only once, then execute a dedicated "repair" step for failing points rather than regenerating the whole thing each time. This "decouple generation from repair" pattern confines the expensive full-context operation to when it's truly necessary.

Architectural Thinking for Production

Looking at these approaches holistically, a more robust production pattern is actually a combination of them:

Decouple generation from repair. Initial generation is a context-heavy operation that justifiably consumes more tokens; repair should be a lightweight, localized, targeted operation. Using different prompt templates and context assembly strategies for each is the most worthwhile improvement to prioritize.

Persist and index context externally. Raw metadata should not be repeatedly transmitted as part of conversation history — it should be treated as an addressable external resource. During repair, the LLM only needs a reference and an error location, plus on-demand retrieval of relevant fragments.

"Minimum sufficiency" for tool output. When MCP tools return results, they should follow a "just enough" principle — only return the information the Agent needs to make its next decision. Structured errors (such as error codes, failing node IDs, expected values) are far more efficient than blocks of plain text.

State over conversation. Driving retry logic with an explicit state machine or state object gives you full control over the token composition of each request, avoiding the hidden costs of unbounded conversation history growth.

Summary

This case reveals a counterintuitive truth in Agent engineering: token optimization is often not about smarter prompts, but about better architectural boundary design. Treating "large-context generation" and "small-scope repair" as two fundamentally different types of operations — combined with externalized retrieval, structured state management, and lean tool output — is what allows you to minimize retry costs while preserving repair quality. For any team running code or data engineering Agents in production, this is a dimension well worth designing carefully.

Background: What Is MCP?

MCP (Model Context Protocol) is an open protocol led by Anthropic, designed to standardize the interaction interface between LLMs and external tools and data sources. Similar to REST API conventions in web development, MCP defines how tools expose capabilities to models and how call results are passed back. In an Agent loop, the return values of MCP tools flow directly into the next LLM context, so their output size has a direct impact on token consumption. A well-designed MCP tool should serialize error information into a compact structure — for example, {"error_code": "TYPE_MISMATCH", "node_id": "transform_3", "expected": "string", "got": "int"} — rather than returning a full error stack trace or raw dataflow. This design requires factoring in "Agent consumption efficiency" during the tool development phase, not just functional correctness.