Webhook Catcher Silently Drops Events: The Hidden Trap of 'Up to 100'

When "Up to 100" Becomes an Excuse for Silently Dropping Data

A developer built a webhook catcher for integration testing: it provides an HTTPS URL, receives POST requests, stores events for 24 hours, and makes them available for later reading. The spec said "stores up to 100 events," so he implemented it with a ring buffer and considered the job done. Then a Stripe test suite burst-sent 112 webhooks—including subscription creation, updates, renewals, and a retry storm caused by an unstable endpoint—and the catcher stored only 100. Events 101 through 112 vanished silently. No error, no overflow flag, no truncation marker of any kind. The developer read the 100 events, assumed the sender had stopped, and shipped a broken integration.

The root cause of this failure wasn't the capacity limit itself—it was that the limit was invisible. A receiver that silently drops data is worse than one that outright rejects it—at least a 429 status code tells the sender to retry.

Why Silent Loss Is More Dangerous Than Rejection

In distributed systems, the design of failure modes directly determines a system's observability and reliability. When a webhook catcher reaches its capacity limit, if it chooses to silently drop data, the caller cannot distinguish between "the sender stopped sending" and "the receiver dropped the data." This ambiguity leads downstream integrations to make decisions based on incomplete data without ever knowing it.

According to the developer, retry storms in Stripe test suites are a common scenario: when an endpoint is unstable, the sender repeatedly sends the same event or subsequent events, spiking the event count in an instant. A ring buffer is efficient, but without an overflow signal, it becomes an accomplice to data loss.

"A receiver silently dropping data is worse than outright rejecting it—at least a 429 tells the sender to retry."

This view resonates with other developers who have built similar systems. In queue and buffer design, boundary conditions must be expressed explicitly; otherwise, a resource limit becomes a hidden threat to data integrity.

Ring Buffer is a fixed-size data structure that uses two pointers (a write pointer and a read pointer) cycling over a pre-allocated array. When the write pointer catches up to the read pointer, new data overwrites the oldest data (or, depending on the implementation, new data is simply discarded). Its advantages are constant memory usage and O(1) operation complexity, making it ideal for temporary storage of streaming data. But this is exactly where the classic ring buffer trap lies: when overwriting or discarding occurs, if the implementer doesn't additionally record the overflow count or set a flag, the caller has no way to perceive from the outside that data has been lost. In the webhook catcher scenario, once the ring buffer fills to 100 events, the 101st event overwrites the 1st, yet the reader still sees what appears to be a "complete" list of 100 events—misjudging data integrity.

HTTP 429 (Too Many Requests) and 507 (Insufficient Storage) are two standard status codes related to capacity. 429 indicates the client has sent too many requests in a given amount of time, usually accompanied by a Retry-After header indicating how long to wait before retrying; 507 indicates the server cannot store the content needed to complete the request, commonly used in WebDAV scenarios but generically applicable to indicate insufficient storage. The key difference between these and silent dropping is that they explicitly signal to the sender that "the data was not accepted," enabling the sender to execute retry, backoff, or alerting logic. In distributed systems, the observability of failures is just as important as success—a receiver returning 429 has temporarily rejected the data, but it preserves the contract of data integrity; a receiver returning 200 while dropping data breaks that contract.

Three Fixes: Making the Boundary "Speak Up"

The developer fixed the problem at three levels, each worth borrowing for similar tools.

1. Overflow as a First-Class Event

Event #101 is no longer discarded; instead, a synthetic record is generated: {type: "overflow", dropped: N}. This way, the reader traversing the event stream will always encounter an explicit overflow marker and know the stream has been truncated. This approach turns "loss" into an observable event rather than hiding it behind a success response.

First-class Event refers to a record that, in the system's event model, is given the same status as other regular events—one that can be uniformly traversed, consumed, and processed. Modeling "overflow" as a first-class event means it is no longer a side-channel log or internal counter, but enters the event stream itself, becoming a record that the reader will inevitably encounter. This design pattern is common in Event Sourcing and Change Data Capture (CDC) systems: for example, when an upstream log is truncated, the system inserts a "gap" or "truncation" marker event so downstream consumers can detect data discontinuity. Its core value is that detectability is built into the data stream—the reader doesn't need to query additional metadata or maintain a separate state machine; simply traversing events with normal logic will trigger alerting or degraded handling when it encounters {type: "overflow"}. This also aligns with the design principle of "make illegal states unrepresentable": rather than making the caller guess whether data is complete, make incompleteness itself an explicit, programmable signal.

2. Read API Returns truncated: true

Alongside the event list, the read API adds a boolean field truncated. Polling agents can detect data loss without manually counting. For automated integration tests, this flag can directly trigger alerting or retry logic, preventing execution from continuing on incomplete data.

Polling Agent refers to a client program that actively makes requests to an API at fixed intervals, pulls incremental data, and processes it. In integration testing and monitoring scenarios, polling agents are often used to simulate real consumers and verify end-to-end correctness of webhook pipelines. The typical logic of a polling agent is: record the last read position or timestamp, fetch new events each time, and advance the cursor. If the receiver silently drops events beyond capacity, the polling agent's cursor advances normally, and it won't notice that a segment of events was never returned. With the truncated boolean field, the polling agent can determine at O(1) cost whether each response is complete, and upon detecting truncation, pause cursor advancement, trigger a retry, or raise an alert. This is far more reliable than having the agent maintain its own event count and compare against an expected total, because the agent often doesn't know how many events the sender actually sent—especially in scenarios where retry storms cause the event count to fluctuate dynamically.

3. Create Response Exposes Capacity Limit

The create-catcher response now includes max_events: 100. Callers know the contract boundary before sending their first POST and can plan their sending pace or adjust their testing strategy accordingly. This transparency turns "up to N events" from an implicit limitation into an explicit agreement.

Lessons for Building Catchers, Queues, and Buffers

This developer's experience is not an isolated case. When building any middleware with capacity limits, boundaries must "speak up." A full buffer should be a signal, not a shrug.

Specifically, designs should follow these principles:

• Make limits visible: Explicitly expose capacity limits in API responses, documentation, and error messages.
• Make loss detectable: Provide overflow events, truncation flags, or count verification so callers can discover data loss.
• Rejection over silence: When unable to store, return explicit errors (such as 429 or 507) to prompt the sender to retry or adjust.

These principles align with best practices for distributed system observability: any resource limit should be accompanied by corresponding monitoring metrics and alerting mechanisms.

Conclusion: The Spirit of Contract in API Design

"Up to N events" is not a storage limit—it's a communication contract. If a tool has a ceiling, that ceiling must be visible in every response—otherwise you're just hiding data loss behind a success status code. For developers building webhook catchers, queues, or buffers, making boundaries explicit is not just a technical choice but a matter of contract responsibility toward callers.

The catcher has been packaged and released, available for webhook receivers that respect their own limits.

One-line Summary

A webhook catcher silently dropped events beyond its limit, causing integration defects and revealing that capacity limits must be explicitly visible.

Paragraph Summary

The article describes a developer who built a webhook catcher and, due to a spec saying "up to 100 events," implemented it with a ring buffer. When a Stripe test suite burst-sent 112 webhooks, it silently dropped 12, causing downstream integration to ship based on incomplete data. The author points out that silent dropping is more dangerous than outright rejection because callers cannot distinguish between a sender stopping and data being lost. The developer implemented three fixes: inserting overflow as a synthetic event into the event stream, returning a truncated flag from the read API, and exposing the max_events capacity limit in the create response. The article then distills design principles for capacity-limited middleware: make limits visible, make loss detectable, and prefer rejection over silence, emphasizing that "up to N events" should be treated as a communication contract that must be visible in every response.