Webhook Always Returns 200? The Silent Payment Callback Loss Trap and How to Fix It

A bug that made the payment flow "falsely succeed"

One morning, I spent most of the day troubleshooting a bizarre phenomenon: the payment flow worked perfectly in staging, but in real tests it never triggered the fulfillment task.

The final conclusion was a bit chilling—my webhook endpoint, upon receiving an event, immediately returned an HTTP 200, and only then slowly went about processing it. Stripe's retry mechanism saw the 2xx, deemed the delivery successful, and would never send it again. Result: the charge succeeded, but the fulfillment task never ran once.

The problem wasn't "a loop written incorrectly somewhere in the code," but rather a fundamental misunderstanding of webhook semantics from the ground up.

"Received" does not equal "processed"

What really made me change the code wasn't moving the processing logic before the response, but accepting a fact: received and processed are two completely different acknowledgments.

In most webhook protocols, the HTTP status code serves as a signal of "was this delivery accepted," not "was the business completed." Conflating these two things leads to endpoints that "look stable but actually swallow events."

The fix is actually quite simple:

• Only return 200 after the event has been persisted and enqueued;
• Any error during processing should throw a 5xx to let the sender retry;
• Decouple queue consumption from the response, so consumer failures don't affect the ack logic.

After the fix, I realized how dangerous things had been: Stripe's retry window is up to 3 days, and my old code had "eaten" the only copy of the event on the very first attempt.

Idempotency is another easily overlooked yet equally fatal concept in webhook integration. Since the sender keeps retrying when it doesn't receive a 2xx, your endpoint may receive the same event multiple times. If the processing logic isn't idempotent—for example, creating a fulfillment record every time a charge.succeeded is received—then retries can actually cause duplicate charges or duplicate shipments. Common approaches include deduplicating at the persistence/enqueue stage using the event ID (like Stripe's event.id) as a unique key, or using an optimistic lock with "query first, then write" at the business layer. In other words, correctly handling retries requires acknowledging that duplicate delivery is the norm, not the exception—this is the flip side of the same coin as the mistaken assumption that "received means processed."

Turning the receiving end into an observable tool

The most painful part of debugging was not knowing how many times the sender had retried, or what headers and payloads each retry carried. You could only guess.

Later, I packaged the receiving end into a standalone tool: webhook_catch.html. Its idea is simple—it gives you a one-time HTTPS URL that can store up to 100 events for 24 hours, so you can truly see: whether the sender retried, when it retried, and what each payload looked like.

The "premature ack" trap I fell into is exactly the kind of problem such tools are best at exposing. When you can see the complete headers and body of every retry, you no longer have to guess why an event "disappeared."

Webhook is a contract about failure

The biggest takeaway from this wasn't a specific fix, but a shift in perspective.

A webhook is not a contract about "delivery," but a contract about "failure." It assumes networks will hiccup, consumers will crash, and messages will duplicate. The sender designs retry logic because it assumes you may fail at any time—and what you need to do is clearly tell it "I didn't finish processing, please send it again" when you fail.

If your endpoint can't distinguish between "I got it" and "I finished processing it," then what you have isn't a webhook integration, but a race condition with a few extra steps.

A few things worth self-checking

1. Is your ack sent before or after enqueuing?
2. When processing fails, do you return 2xx or 5xx?
3. Is the event handled idempotently to cope with the sender's multiple retries?
4. Do you have a way to observe the raw payload of "the Nth retry"?

If you can't answer even one of these four questions, events may already be silently disappearing in production.

In distributed systems, this type of problem is called at-least-once delivery semantics—the sender guarantees an event is delivered at least once, but not that it's delivered only once. Its counterparts, at-most-once (may lose) and exactly-once (theoretically optimal, extremely hard in engineering), make up the three tiers of message delivery semantics. Almost all mainstream webhook providers (Stripe, GitHub, Shopify, etc.) choose at-least-once: they'd rather you receive duplicate events than miss a critical notification. Once you understand this, accepting that "retries are the norm and idempotency is an obligation" is no longer a passive compromise, but alignment with the explicit contract of the protocol designers.