The Idempotency Key Problem: How to Handle 'Zombie Processes' in API Retries

Where It All Starts: Why Sending an Email Isn't Idempotent

Retry logic in APIs looks simple on the surface, but it hides some serious pitfalls. A Reddit developer shared a classic dilemma they ran into with an email-sending endpoint: sending an email is not an idempotent operation. You only know the email was actually delivered once the email provider accepts the request and returns a message ID.

This means if the caller's request times out, they have no way of knowing whether the email was sent or not — retrying risks sending a duplicate, but not retrying risks missing the send entirely. This tension shows up everywhere: payments, notifications, message delivery. It's one of the unavoidable challenges in distributed systems.

Idempotency means that performing the same operation once or multiple times produces exactly the same result. In HTTP semantics, GET, PUT, and DELETE are defined as idempotent methods, while POST generally is not. Sending an email is a classic "side-effectful external call" — each invocation may trigger a new email delivery action, and the server has no way to automatically determine whether two requests with the same content represent a retry or a new request. Contrast this with a database INSERT ... ON CONFLICT DO NOTHING or a payment system that charges a given order only once — these operations support idempotency naturally through internal state checks. Email, SMS, and Webhook delivery are tricky precisely because their side effects happen outside the system boundary, and callers can't safely retry without introducing additional state.

The Existing Approach: Using Idempotency Keys to "Claim" Execution

This developer's approach is quite representative and worth breaking down.

Insertion as a "Claim"

Before the actual business logic (the handler) runs, a record is inserted for each Idempotency-Key. This table has a unique constraint on three fields: org + key + endpoint. This insertion is essentially a preemptive claim — whoever inserts first wins the right to execute that request.

Handling Three Retry States

Based on this claim mechanism, retry requests fall into three clearly defined categories:

• Retry while in-progress: If the first request is still running, the retry hits the unique constraint and receives a 409 Conflict, explicitly telling the client "still processing, please wait."
• Retry after completion: If the request has already finished, the retry simply returns the previously stored response, preserving idempotent semantics.
• Expiration and cleanup: These records are automatically purged after 24 hours to prevent unbounded data growth.

This design is clean and effective. It pushes the "deduplication" responsibility to the front, relying on the database's unique constraint — simple to reason about and easy to implement.

The Unsolved Core Problem: Dead Holders

The truly tricky part is when a process crashes mid-call — the "dead owner" scenario.

When a process successfully inserts an idempotency key and then crashes just as it's about to call the email provider, that key gets stuck. Because the record already exists, every subsequent retry hits a 409, while the actual email may never have been sent. The result: the key remains occupied until the 24-hour TTL expires. During this window, the operation is effectively frozen.

An Uncomfortable Trade-off

The developer also proposed a candidate solution: using a shorter lease with a renewal mechanism. The holder periodically renews the lease; if renewals stop, the holder is assumed dead and the lock is released.

But this introduces a new risk: if the liveness check makes a wrong call — say, the original process was just experiencing a network blip, a GC pause, or a brief stall rather than truly dying — then prematurely releasing the lock lets a new process take over and send the email again, causing a double send.

This is the classic dilemma in distributed locking: set the lease too long and fault recovery is slow; set it too short and the false-positive risk is high. You can't easily have both.

This dilemma is academically related to the "Two Generals Problem" — a variant of fundamental distributed systems challenges, also connected to the FLP impossibility theorem: in an asynchronous network, you cannot reliably distinguish between "node is down" and "node is slow." The famous debate between Antirez (author of Redis's Redlock algorithm) and distributed systems researcher Martin Kleppmann centers precisely on whether short leases can cause lock safety failures under network partitions or process pauses (such as JVM GC Stop-the-World events). For scenarios like email delivery, even NTP-synchronized physical clocks cannot fully eliminate the judgment errors introduced by clock drift — this is a theoretical ceiling of the lease mechanism, not an implementation bug.

Some Possible Directions

While the original post doesn't offer a definitive answer, common distributed systems practices suggest a few directions worth exploring:

Make the Downstream Idempotent

If the email provider supports client-specified idempotency keys (some major providers do), then even if a double send occurs, the downstream can detect and deduplicate it. This essentially shifts idempotency responsibility one layer down — eliminating the consequences of duplicate sends rather than relying solely on the upstream lock being perfect.

Major payment and email providers like Stripe, SendGrid, and Postmark all offer idempotency key support at the API level. With Stripe, for example, passing Idempotency-Key: <uuid> in the request header causes the server to return the cached original response for 24 hours for the same key, without re-executing the charge. SendGrid doesn't have native idempotency keys, but supports limited deduplication via custom headers like X-SMTPAPI with a unique send_at and batch ID. This "push idempotency downstream to the provider" approach is the option most worth evaluating first when building reliable message delivery systems — it transforms a complex distributed coordination problem into a matter of reading API documentation.

Explicitly Choose Between "At Most Once" and "At Least Once"

For scenarios like email, you need to first decide which failure mode the business can tolerate. If a missed send is worse than a duplicate send, you should lean toward more aggressive retry on dead-holder detection, paired with downstream deduplication. If the opposite is true, be conservative. Engineering decisions should serve the business's fault-tolerance preferences.

Record Execution Phase State

You can add a status field to the idempotency record to indicate what phase the request is in: "claimed but not sent," "sent but unconfirmed," "completed," and so on. When a holder crashes, the takeover process can use this state to determine whether it's safe to retry, rather than blindly re-sending or blocking.

Summary

This real-world problem captures the essence — and limitations — of idempotency design. Using a database unique constraint for preemptive claiming is an elegant engineering technique, but it cannot solve the fundamental problem of process crashes on its own.

Truly robust solutions rarely rely on any single layer being perfectly reliable. Instead, they distribute risk across layers — upstream claiming and deduplication, downstream final deduplication — absorbing failures through collaboration. For any engineer building high-reliability APIs, understanding and weighing these trade-offs matters far more than reaching for an off-the-shelf pattern.