Stop Dumping Raw Data Into Your Prompts: The Real Fix for AI Agent Context Bloat

A few months ago, I watched an agent do something baffling. It had already fetched the data. It had already parsed it. It had already stored the useful fields. And then it went ahead and pasted the entire raw tool response, byte for byte, into the next model call.

The full JSON blob, the logs, the metadata, the useless fields — all of it. The next step got slower. The one after that got more expensive. Then the model started doing that infuriating thing where it only answers the easy part of the task and quietly ignores the important part.

That was the moment the bug became obvious: most so-called "context window problems" aren't model problems at all. They're state management problems. Once you see it, you start spotting the same mistake everywhere — in n8n flows, LangGraph agents, MCP tool loops, and all kinds of custom OpenAI-compatible orchestration code.

The root cause: treating chat history like a database

A lot of people conflate two ideas: the model can hold a long context, and the model should receive a long context. These are completely different things.

A giant prompt is really just badly managed state storage disguised as a conversation. Even if the model can technically accept the full history, performance still gets mushy as stale content piles up. Latency climbs. Cost climbs. Attention gets diluted.

This is why long-running agents often look smart for the first 5 steps and then become mysteriously unstable by step 20.

The core fix: stop replaying raw tool output

The real fix is almost embarrassingly simple: keep the full result in structured state outside the model, and send only one of three things — a short summary, the exact fields the next step needs, or a precise evidence slice when verification matters.

This single change usually improves agent quality more than swapping from one frontier model to another.

The comparison across three patterns makes it obvious. Cram raw tool output into the prompt, and the prompt grows fastest, adds the most noise, and latency and cost climb with every step. Summarize tool results into the prompt, and you usually get a decent balance — though a bad summary can hide edge cases. Keep structured state outside the prompt, and you get the best default for multi-step agents: a small, repeating prompt, cleaner reasoning, and better orchestration.

The wrong way

The pattern below dumps the entire API response straight into the prompt. If result contains 200 lines of hourly forecast data, provider metadata, timestamps, debug fields, and tracking IDs, the prompt instantly becomes a junk drawer. The model has to find the relevant fields, ignore the irrelevant ones, remember prior steps, plan the next action, and avoid getting distracted by noise. The model can do it. It's just inefficient.

The right way

Store the full result in application state, and pass only the fields the next step needs. Same workflow, smaller prompt, less noise, and the model is far more likely to focus on the decision itself rather than re-parsing garbage data.

What goes where

Here's the mental model I wish more agent builders started with: short-term conversational state lives on the thread. Long-term facts and tool results live in a database or memory store outside the prompt. The next model call gets only the smallest useful slice of that state. It's not complicated. It just takes discipline.

How real frameworks handle it

LangGraph: checkpointed state, not replayed conversation

LangGraph gets this right. Instead of shoving prior tool output back into every prompt, it persists graph state and resumes by thread. That's the grown-up version of memory. The model gets what the next node needs, and Postgres keeps the rest.

n8n: deliberately trimmed memory

n8n's approach here is pretty pragmatic. If you're building AI workflows in n8n, you already have the tools to avoid context bloat: Simple Memory, Redis, Postgres, Xata, Zep, Motorhead, Chat Memory Manager. The fact that the Chat Memory Manager node exists says it all — if a workflow platform ships a dedicated node for inspecting and reducing memory, then "always append to the chat log" clearly isn't the best pattern.

MCP: tool output is application state first

MCP makes this cleaner because tools are structured to begin with. Tool definitions include inputSchema and outputSchema, which should push you toward a simple pattern: call the tool, store the full structured result, and pass only the fields you need into the next prompt. The prompt becomes a working note, not a warehouse.

MCP (Model Context Protocol) is an open standard released by Anthropic in late 2024, designed to unify how large language models interact with external tools and data sources. Before MCP, every agent framework or application had to define its own tool-calling format and lifecycle, which fragmented the tooling ecosystem; MCP abstracts "how a tool is described, how it's called, and how it returns results" into a standard interface. Tools declare which parameters callers can pass via inputSchema, and declare the structure of returned results via outputSchema. The point of these explicit schemas isn't just type validation: they naturally tell the caller "which fields are structured and can be extracted individually," making "store the full result in state and pull out only the fields the next step needs" a natural operation rather than something the developer has to manually dig out of a blob of text.

Understanding the difference between "persisted state" and "conversation history" is the key to this section. Conversation history is a chronologically appended sequence of messages that the model has to read from start to finish every time. Persisted state is a structured key-value container that allows reading, updating, and deleting arbitrary fields on demand. LangGraph's checkpointer is the latter: each thread corresponds to a recoverable snapshot of graph state, and a node reads only the fields it cares about instead of re-feeding all prior messages to the model. n8n's various Memory nodes (like Chat Memory Manager) offer similar capabilities, letting you actively trim, summarize, or replace memory within a workflow. MCP defines tool results themselves as structured objects, making them naturally suited to being part of state. The common thread across all three is: state is where facts live, and the prompt is just "the minimum view needed for this step's reasoning."

Prompt caching won't save bad context hygiene

This one gets misunderstood a lot. Anthropic's prompt caching is useful when you're reusing the same stable prefix — system instructions, tool definitions, fixed documents, repeated scaffolding. But it can't magically make a sloppy prompt good.

Claude caching only kicks in when the prefix is exactly identical, and cache entries only last 5 minutes by default. That's great for repeated prefixes, but it won't make a giant raw tool dump any more relevant. If anything, caching can mask bad architecture for a while, because the bill stops hurting before the quality problem surfaces.

Use caching to optimize stable prefixes. Don't use it as an excuse to keep stuffing junk into the prompt.

The core mechanism of prompt caching is "prefix matching": the server caches a stable chunk of content starting from the beginning of the prompt, and the next request with a byte-for-byte identical prefix can skip recomputing that part, reducing latency and cost. Anthropic's cache lasts 5 minutes by default, and each hit refreshes the timer; it requires the prefix to be exactly identical, and any change invalidates the cache. This means caching is best suited to "long prefixes that barely change," such as system prompts, tool definitions, and fixed reference docs. But it can't help with "content that keeps bloating in the middle or back of the conversation" — raw tool output tends to appear mid-to-late in the conversation and differs each time, so it can't hit the cache at all. More subtly, caching lowers the billing cost of redundant prefixes that could have been avoided, temporarily numbing developers to the pain and thus delaying the real fix to their context architecture.

Before and after the refactor

In a typical agent loop, the old version looked like this: search documents and CRM data were formatted directly into the prompt with full content. It worked at first, then search_results grew, crm_data grew, and every step dragged more baggage.

After the refactor, the full search results and CRM data go into state, and the prompt contains only the user request, the top three document titles, customer tier, open ticket count, and billing status. This version is far less error-prone. When the model later needs exact evidence, it can selectively pull the raw state back.

When to keep raw output

There are real exceptions. Sometimes the next step genuinely needs the exact source material. When you need legal or compliance citations, exact stack traces for debugging, code review of a full diff, extraction from messy text, or verification against raw evidence, you should keep the raw output in context.

But even in those cases, pass only the exact slice that matters, not the entire upstream universe.

The rule I use now

This heuristic cleaned up most of my agent workflows: if the information is for reasoning, summarize it. If it's for retrieval, store it. If it's for verification, pass the exact slice.

This rule has saved me more latency, cost, and agent quality than any model swap ever has.

When you pay per token, this matters even more

This gets expensive fast in production. If your agent runs 24/7 in n8n, Make, Zapier, or a custom loop, every extra chunk of data you replay becomes a recurring cost. It's also one of the reasons many teams end up obsessing over token usage dashboards instead of fixing their architecture.

Cleaner state design reduces prompt bloat at the source. It's even better if you're using an OpenAI-compatible endpoint with flat monthly pricing, because then you can optimize for agent quality and throughput instead of constantly worrying about whether one more tool call will spike the bill.

The author specifically mentions Standard Compute, which works as a drop-in replacement for the OpenAI API, letting you keep your existing SDK and agent code while no longer thinking in "per-token panic" mode. The architecture fix still matters — you just get to implement it without token anxiety.

A practical checklist

If your agent feels slower and dumber over time, check these first: are you pasting raw tool output into every subsequent prompt? Are you storing canonical results outside the model? Are you summarizing tool results before the next reasoning step? Are you passing exact evidence only when verification is needed? Are you using prompt caching only for stable prefixes? Are you trimming memory in n8n or your orchestration layer? Are you treating the model as a reasoning engine rather than a database?

Just fixing these will likely give you more improvement than running yet another model comparison test.

I went looking for a model-level fix and found an architecture-level one instead. Most "context window problems" are really just us treating chat history as a storage layer. Once you stop stuffing raw tool output back into the next prompt, agents get faster, cheaper, and noticeably less confused. Not because the model changed, but because we finally stopped making them carry our state management mistakes.

One-line Summary

Most "context window problems" are really state management problems — keep raw tool output outside the model and they go away.

Paragraph Summary

The article argues that many agents degrade not because of model capability, but because we treat chat history as a database and repeatedly replay raw tool output. The author advocates storing full tool results in structured state outside the model and passing only a short summary, the fields the next step needs, or a precise slice for verification. LangGraph, n8n, and MCP already offer paths to this via checkpointed state, memory trimming, and tool schemas respectively. The article also clarifies that prompt caching only helps with stable prefixes and can't compensate for poor context hygiene — it may even mask architectural flaws. The core rule: summarize information used for reasoning, store information used for retrieval, and pass exact slices for information used for verification.