Chatbots Resend Full History Every Turn: Real-World Context Costs and Compression Strategies

Chat APIs are stateless. To answer the 41st message, your app has to resend the previous 40 messages — and all of that history is billed as input tokens. Every message gets paid for again.

[image]

This isn't a trivial amount. According to an analysis based on real API traffic, across roughly 411,000 requests over 7 days, 99.3% of processed tokens were input tokens; the median request had 3,782 input tokens and returned only 147 tokens. The real bulk of most LLM bills isn't long answers — it's long history.

A Real Side-by-Side Experiment: Three Context Strategies

To quantify the cost of "how much history to keep," the author designed a customer-service bot scenario: after 40 turns of conversation, ask one final question, and compare three strategies:

Full history: what most tutorials do
Last 6 turns: sliding window
Summary + last 4 turns: use a cheap model to compress earlier conversation into a 60-word note

The test script trim_context.py uses the claude-sonnet-5 model, with input pricing at $0.3835/1M and output pricing at $1.5339/1M. The three strategies send different messages, but the model and output length stay consistent.

The measured results are as follows:

Strategy	Input tokens	Cost per message	Relative to full history
Full history (40 turns)	3,690	$0.001599	—
Last 6 turns	596	$0.000413	−74%
Summary + last 4 turns	531	$0.000388	−76%

Same model, same output length, and the bill drops to about a quarter. The summarization step itself has extra overhead: one call to GLM 5.3 Flash costs about $0.00002, and you only pay it when history exceeds your set limit, not on every message.

Stateless means the API itself stores no session memory: each request is independent, and the server doesn't "remember" what you said last time. So to support multi-turn conversation, the client must include the prior user and assistant messages in this request's messages array. This is also the root cause of history tokens being billed repeatedly — the same history gets resent as input every turn. Some platforms offer managed "conversation state" capabilities (such as OpenAI's Responses API or Assistants' state storage), where the server stores history, but the billing logic is usually still based on the input tokens actually processed. Understanding this mechanism helps you identify where cost optimization should focus.

Cheap Comes at a Cost: Sliding Windows Develop "Amnesia"

Cost is only half the story; the other half is quality. The author planted a fact at the start of the conversation: "My customer is Northwind, account ID 7731, net-45 terms." Then, after 40 more turns, the bot was asked to recite this ID and the terms, running each strategy 3 times:

Strategy	Input tokens	Does it remember "7731, net-45"?
Full history	3,751	✅ 3/3
Last 6 turns	600	❌ 0/3 ("I haven't received any account ID")
Summary + last 4 turns	~530	✅ 3/3

A pure sliding window is the cheapest, but it forgets everything outside the window — including key facts the user mentioned only once. The summary approach preserves those facts at roughly the same price. To improve reliability, explicitly require it in the summary prompt: "Keep names, IDs and terms."

15 Lines of Production-Grade Code: Summary + Window

The author provides a pattern you can put into production directly: keep MAX_TURNS = 8 turns of verbatim text, and when history exceeds SUMMARY_AT = 12 turns, hand the earlier portion to a cheap model to compress into a summary, then send the summary together with the recent turns.

The core function build_messages(system, turns, summary, new_msg) works like this:

When len(turns) > SUMMARY_AT, call summarize() to merge turns[:-MAX_TURNS] into the existing summary, and keep only the most recent MAX_TURNS turns verbatim;
The message array starts with the system prompt, and if a summary exists, append a system message Earlier: {summary};
Finally, append the retained turns and the new message.

Key engineering detail: store the summary alongside the conversation, so the same history is compressed only once and never paid for repeatedly.

MAX_TURNS = 8         # keep this many recent turns verbatim
SUMMARY_AT = 12       # compress once history passes this

def build_messages(system, turns, summary, new_msg):
    if len(turns) > SUMMARY_AT:
        summary = summarize(summary, turns[:-MAX_TURNS])  # cheap model
        turns = turns[-MAX_TURNS:]
    msgs = [{"role": "system", "content": system}]
    if summary:
        msgs.append({"role": "system", "content": f"Earlier: {summary}"})
    return msgs + turns + [new_msg], turns, summary

A sliding window is the simplest history-trimming strategy: keep only the most recent N turns and discard everything earlier. Its advantages are simple implementation and controllable token usage; its drawback is that information outside the window disappears entirely, and the model can no longer reference it. In contrast, a rolling summary hands old content beyond the window to a cheap model that compresses it into a short note, which is continuously updated and persisted across the conversation. Since the summary is only generated when new history overflows, the compression cost is amortized, and key facts (names, IDs, terms) can be preserved by explicitly requiring them in the summary prompt. The two are often used together: recent verbatim text ensures detail accuracy, while distant summaries ensure long-term memory.

Other Input Token Leaks Worth Investigating

Besides conversation history, there are several other common "invisible input waste" points:

System prompts bloated to 2,000 tokens: sent on every call; they should be trimmed, or you should use a provider that supports prompt caching.
RAG stuffing 10 chunks when 3 would do: rerank first, then trim.
Tool schemas: every function definition is input tokens on every call; send only the tools actually needed for the current step.

The author's traffic data comes from Qubax — an OpenAI-compatible API service that aggregates 400+ models; the trimming code in the article can be used with any OpenAI-compatible endpoint.

Prompt caching is an optimization mechanism provided by model providers: when multiple requests share the same prefix (such as a fixed system prompt or the initial part of a conversation history), that prefix is only fully billed and processed on the first request, and subsequent requests can hit the cache, reducing the price of input tokens and lowering latency. OpenAI, Anthropic, and others all offer this capability; it usually requires explicitly marking cache boundaries, and caches have a time to live (e.g., 5 minutes or 1 hour) and must be rewritten after expiry. Note that caching only makes repeated prefixes cheaper; it does not shorten the total context length, so long histories still occupy the context window and increase latency. Best practice is to combine trimming with caching.

Reranking is a step in the RAG pipeline: initial retrieval usually recalls many chunks by vector similarity (e.g., 10-20), but these chunks aren't necessarily the most relevant to the question. A reranking model (a cross-encoder or a dedicated reranker) reads the question and candidate chunks together, outputs a more precise relevance ordering, and then the few highest-scoring chunks are passed to the LLM. Because reranking happens before the LLM call, it can significantly reduce the number of tokens stuffed into the context while improving answer quality. A common approach is to recall 10-20 chunks and keep only 3-5 after reranking.

FAQ

Is a sliding window good enough?

It's enough for short-lived conversations. But in the test, it forgot the account ID given at the start of the conversation 3/3 times. If the user states a fact once and expects it to be remembered, you need to summarize earlier turns.

Which model should I use for summarization?

A cheap, fast model. In this test, a summarization call cost about $0.00002, and you only pay it when history exceeds the limit.

Can prompt caching solve this problem?

Partly. Caching makes repeated prefixes cheaper, but long history still increases latency and may push you past the context limit. Trimming and caching work best together.

References

OpenAI managing conversation state: https://platform.openai.com/docs/guides/conversation-state
Anthropic prompt caching: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
OpenRouter model pricing: https://openrouter.ai/models

One-line Summary

Stateless chat APIs cause history tokens to be billed repeatedly; summaries plus a sliding window cut costs by 76% while preserving key facts.

Paragraph Summary

Chat APIs are stateless, so every turn resends the full history. In measurements, 99.3% of processed tokens were input tokens, meaning the bill is driven by history, not answers. The author ran a 40-turn customer-service bot experiment: full history cost $0.001599 per message, a last-6-turns sliding window dropped it to $0.000413 (−74%), and summary plus last 4 turns dropped it to $0.000388 (−76%). But a pure sliding window forgot the account ID and terms given at the start in 3/3 tests, while the summary approach remembered them all at nearly the same price. The production pattern is to keep the last 8 turns verbatim, compress earlier history with a cheap model once past 12 turns, and persist the summary so it's compressed only once. System prompts, RAG chunks, and tool schemas are also common input leaks and should be trimmed or combined with prompt caching.