Two Hidden Pitfalls in LangChain Caching Gateways: Tool Call Ordering and Streaming Responses

The Origin: Complexity Behind a Single Line of Code

In LangChain, pointing ChatOpenAI's base_url to a custom gateway seems to require just one line of code. But when you try to implement request caching at the gateway layer, you quickly discover that LangChain's request structure is far less stable than it appears.

While building the Metrecept caching gateway project, developers ran into two classic but easily overlooked technical obstacles. The root cause lies in the impedance mismatch between the framework's abstraction layer and the underlying protocol.

Pitfall #1: Non-Deterministic Tool Call Ordering Breaks Cache Keys

When using LangChain's AgentExecutor to run tasks that involve multiple tool calls, the serialized tool-call list order can vary between runs — even when both runs are functionally identical. This causes hash-based cache keys to treat them as different requests, resulting in a high rate of cache misses.

This problem is especially pronounced in retry scenarios. Developers initially noticed unusually low cache hit rates for requests involving tool-call retries. Investigation revealed the culprit: the JSON serialization order of tool calls. Python dictionary iteration order is not always stable across certain execution paths, meaning the same set of tool calls can produce different request bodies.

The fix is to apply a canonical sort to the tool call list before generating the cache key. However, this requires the caching layer to have a deep understanding of LangChain's request structure — you can't just treat the HTTP request body as a black box.

Pitfall #2: The Architectural Challenge of Caching Streaming SSE Responses

The streaming API (.stream()) in LangChain assembles responses chunk by chunk on the client side, which creates an architectural challenge for server-side caching: the caching gateway must fully receive and reassemble all SSE (Server-Sent Events) chunks on the server side before it can store the complete response in the cache — and only then can it replay the response as synthetic SSE for subsequent identical requests.

This means partial streaming responses will never become cache entries. If the client disconnects mid-stream, or if the LLM generation is interrupted abnormally, the caching layer will not save the incomplete response. While this guarantees the integrity of cached content, it also increases memory pressure and latency at the caching layer — the system must wait for a complete response before deciding whether to cache it.

Design Trade-off: Exact Match vs. Semantic Similarity Match

The project deliberately chose exact matching over semantic similarity matching. The reason is straightforward: introducing fuzzy matching for "nearly identical prompts" at the caching layer is fundamentally trading accuracy for performance, and risks returning incorrect answers that the user never actually requested.

This design decision reflects a core principle: the caching layer should be a transparent acceleration component, not an intelligent agent with its own judgment. Once a cache starts "understanding" the semantics of requests and making similarity judgments, it introduces new sources of non-determinism and debugging complexity.

The Underlying Issue: The Cost of LangChain's Abstraction

The root of these problems isn't LangChain itself — it's the impedance mismatch that arises when you insert a middleware layer beneath the framework's abstraction. LangChain abstracts the underlying OpenAI API format into a higher-level interface, but this abstraction introduces several side effects:

• Request serialization does not guarantee idempotence: Requests with identical logic may produce different JSON structures
• Streaming control flow is distributed: Responsibilities are split between the client and the server
• The caching layer must understand framework internals: Being HTTP-protocol-compatible is nowhere near sufficient

For developers building similar infrastructure, this case reveals a critical insight: when inserting caching, monitoring, or proxy layers into an AI application stack, you must deeply understand the actual request patterns of the upstream framework — not assume that "OpenAI API compatibility" is enough.

Open Source Project and Community Discussion

The Metrecept project has been open-sourced on GitHub under the MIT license, and the core cache key normalization logic is available for other developers to reference and reuse. The question the author raises is also worth reflecting on: have other teams encountered similar issues when caching LangChain traffic, or discovered additional pitfalls not mentioned here?

Sharing this kind of engineering experience is invaluable to the AI infrastructure community — many seemingly simple "just add a proxy layer" solutions often hide subtle problems that only surface in production.