Cache Key Defects in an Agent PR: Cross-Tenant Leakage and Test Verification

Title

Cache Key Defects in an Agent PR: Cross-Tenant Leakage and Test Verification

Meta Description

A deep dive into three hidden cache defects in an Agent-generated PR: cache keys ignoring tenant identity leading to cross-tenant data leakage, failed results being cached, and a process-local unbounded dictionary. Includes a ten-minute reproducible test plan and a fix decision table.

Article Content

At two in the morning, a PR was submitted with a green pass mark, touching only three files. Its claims were restrained: skip duplicate model calls, reduce latency for repeated prompts. The reviewer's first instinct was to trust the test suite. But the tests only covered the happy path, and that is precisely where cache changes are least likely to go wrong.

This article uses this Agent-generated PR as an example to dissect three hidden cache defects and provide a fix that is reproducible and verifiable within ten minutes.

The Diff Under Review: Obvious Problems at a Glance

The core code is short: a module-level CACHE dictionary, and a complete() function that accepts prompt, tenant_id, and model parameters. The cache key is the SHA-256 hash of prompt. On a hit, return; on a miss, call the model and write to the cache.

Three issues surface on first read. The tenant_id parameter is accepted but never used; there is no check to determine whether the response is usable before storing it in the cache; the dictionary has no size limit, no TTL, and no eviction path.

This diff passed review because it is small and readable. But small and readable does not equal correct.

Why the Happy Path Test Passes

The accompanying test calls complete() twice with the same prompt and the same tenant, asserting that the model client was called only once.

This assertion is correct. And it remains correct after fixing all the defects below. That is why this test suite offers no signal to the reviewer—it never touches the places where the cache system is truly fragile: boundary conditions, failure paths, concurrency scenarios.

Three Defects the Existing Tests Never Touch

1. Cache Key Ignores Tenant Identity

The cache key is merely a hash of the prompt text. Two tenants sending the same string share the same cache entry.

For prompts like "Summarize the incident report.", collisions are not a theoretical possibility. Support tools, templates, and system prompts are repeated across accounts every day. In a multi-tenant product, this means Tenant A can read Tenant B's response content—a direct cross-tenant data leak.

2. Blank and Failed Results Are Cached

If call_model returns None or an empty string due to a transient upstream failure, this value is stored as the answer. All subsequent callers will read this failure result until the process restarts.

Negative caching turns a one-second glitch into hours of downtime. This diff never distinguishes good answers from bad ones.

3. The Cache Is Process-Local and Unbounded

Four workers mean four independent dictionaries and four independent hit rates. Memory grows with the number of unique prompts and never shrinks.

This diff looks like a shared cache but behaves like a per-worker notebook with no eraser.

What to Keep, What to Revert, What to Test

Faced with such a PR, the decision can be summarized in five points:

1. Keep the pure function. hashlib.sha256 is a reasonable key primitive.
2. Keep the tenant_id parameter and incorporate it into the key computation.
3. Revert the module-level mutable dictionary.
4. Revert any writes of empty, failed, or incomplete responses.
5. Replace the single happy-path test with isolation, failure, and concurrency tests.

A Ten-Minute Runnable Reproduction Plan

Step 1: Write the Isolation Test First

The core of the test is to create a fake call_model that records the tenant_id passed in on each call. Then call complete() with the same prompt for two tenants, "acme" and "globex", asserting that both calls reached the model layer and returned different values.

Run pytest -k tenant -x, and it will fail. This failure is evidence, not opinion.

Step 2: Fix the Key and Re-run

The fix is to concatenate tenant_id, model, and prompt with a null byte separator before hashing. The null byte separator prevents ambiguous concatenation. The same test should pass without modification.

Step 3: Prove Failures Are Not Cached

Use monkeypatch to make call_model throw a TimeoutError, call complete() twice in succession, and assert that the underlying call was made twice. If the exception were cached, only one call would occur.

Step 4: Measure Cross-Worker Hit Rate

Send forty requests with the same prompt to the local endpoint and count the occurrences of model_call in the logs. If the count is close to forty, the cache is not shared between workers.

Decision Table: Verdict on Each Change

Change	Verdict	Rationale
cache_key helper function	Keep	Small, pure, easy to test
Include tenant_id in the key	Keep and enforce	Blocks cross-tenant reads
Module-level CACHE dictionary	Revert	Process-local, unbounded, no TTL
Cache empty or failed results	Revert	Turns glitches into persistent failures
Single happy-path test	Replace	Never covers isolation or failure scenarios

Reusing This Review Loop

One detail: the Agent that generated this diff runs on MonkeyCode's free model access. The reproduction suite above runs on the project's free server option, sufficient for hundreds of test calls in a single session.

The operator states that the free tier also includes promotional token credits, currently advertised as 10 million tokens. Confirm the current terms in the repository before planning around any number. Disclosure: This article was prepared as part of a MonkeyCode product promotion.

The model is not the point here. The loop is: generate, reproduce, revert, retest.

Limitations: Who Should Skip This Approach

Several types of teams need extra caution. Teams with strict data residency requirements should not send prompts to any hosted endpoint, free or paid. Cache keys for multi-tenant products must be derived from tenant, model, and prompt version together. Process-local dictionaries are only suitable for single-worker prototypes and are wrong in any other deployment. If hit rate matters at scale, use shared storage with TTL, eviction policies, and size limits.

The review in this article does not attempt to verify model output quality; it only checks whether one tenant can read another tenant's answers.

Conclusion

Agent PRs are drafts, not decisions. Trust pure functions, revert shared mutable state, and test tenant boundaries before doing anything else. The value of this review loop is that it turns "I think there's a problem" into "run this test and watch it fail." In a world where Agents generate code more and more frequently, this transformation from intuition to evidence is more important than any single cache fix.