Simulating Connection Pool Starvation in Async Architectures: Queueing Delay and Leaks

When distributed systems hit traffic spikes, connection pool exhaustion often triggers cascading timeouts. Many engineers' first instinct is to increase max_connections, but this only shifts the pressure onto the database and can lead to kernel resource depletion. To tune scientifically, you must first answer: at what concurrency level does the pool exhaust? How long will requests queue before timing out? How much damage can a single "rogue task" that fails to release a connection cause?

This article is based on a complete Python async testing tool. It breaks down the mechanics of connection pool starvation and demonstrates how to quantify the impact of queueing delay, timeouts, and connection leaks without touching a production database.

How Connection Pool Starvation Happens

A connection pool is essentially a finite set of network sockets protected by a mutex. When concurrent requests exceed pool capacity, excess coroutines queue up in the event loop. If a task exits with an exception while holding a connection and fails to release its slot, the pool capacity is permanently reduced—this is a connection leak.

The diagram below shows the typical flow: high-concurrency requests attempt to acquire a semaphore; if the pool is full, requests enter a waiting queue; waiting beyond a threshold throws a timeout; the normal path returns the slot via try/finally; while a path that bypasses release leaves the slot permanently locked.

flowchart TD
    subgraph Clients ["Client Request Pipeline"]
        R1["High-concurrency client requests"]
    end
    subgraph Pool ["Async Connection Pool"]
        Sem["asyncio.Semaphore: slot allocation"]
        Active["Active slots checked out"]
        Queue["Queue of coroutines waiting for slots"]
    end
    subgraph Outcomes ["Request Outcomes"]
        Success["Task complete: released"]
        Timeout["asyncio.TimeoutError: fast fail"]
        Leak["Exception handler: connection leak"]
    end
    R1 -- "Acquire slot" --> Sem
    Sem -- "Slot available" --> Active
    Sem -- "Pool exhausted" --> Queue
    Queue -- "Wait timeout" --> Timeout
    Active -- "try/finally executes" --> Success
    Active -- "Bypass release" --> Leak
    Success -- "Return slot" --> Sem
    Leak -. "Permanently locked" .-> Active

The key insight is: leaks don't need to happen frequently. The original experiments show that even a 5%–10% release omission rate is enough to gradually reduce pool capacity to zero under moderate traffic.

Accurately Replicating Pool Behavior with asyncio.Semaphore

Production-grade connection pools (asyncpg, HikariCP, SQLAlchemy QueuePool) are all built on semaphores at their core. The original article chose asyncio.Semaphore(max_size) as the simulation foundation because it accurately reproduces three states:

• Permits available: Callers immediately get a slot with minimal scheduling overhead.
• Zero permits: New callers suspend but don't block OS threads—they only register a wait on the event loop.
• Timeout control: Wrap the acquire action with asyncio.wait_for(semaphore.acquire(), timeout) to replicate checkout_timeout behavior and prevent coroutines from starving indefinitely.

The simulator maintains an internal active_connections counter and uses asyncio.Lock to ensure coroutine safety. acquire returns a boolean indicating whether a slot was obtained within the timeout; release accepts a leaked parameter—when True, it deliberately skips semaphore.release(), permanently reducing pool capacity.

This abstraction strips away the complexity of specific database drivers, letting engineers focus on the concurrency control logic itself.

Difference Between asyncio.Semaphore and OS Semaphores

asyncio.Semaphore is a coroutine-level synchronization primitive, not an OS thread semaphore. Its counter is managed single-threadedly by the event loop. When the count is zero, acquire() doesn't block the thread—it suspends the current coroutine into a wait queue and yields control to other runnable tasks. This means even with hundreds or thousands of coroutines waiting for pool slots, only memory and event loop scheduling overhead are consumed—unlike thread blocking, which can exhaust the thread pool or trigger context-switch storms.

Its release semantics are "increment the count and wake one waiter," but release() itself does not verify whether the caller ever held a permit—meaning erroneous or missing calls won't immediately raise an error, making leaks stealthy. In production connection pools, this characteristic is compensated by the driver layer with additional state tracking (e.g., recording checkout owner identity, timeout reclamation), while the simulator exploits this loose semantics to artificially create leaks.

Connection Leak Simulation: try/finally Is Not a Panacea

In real incidents, leaks rarely occur on the "happy path." They often stem from uncaught exceptions interrupting the execution frame before release logic runs. The original worker uses a try/except/finally structure: normal business logic runs in try; exceptions are classified and caught; finally guarantees pool.release() is called.

However, the leak_probability hook deliberately passes leaked=True in finally, bypassing the semaphore release. This is not encouraging bad code—it's to prove a counterintuitive fact in a controlled environment: even with finally protection, as long as there's a bypass path (e.g., an exception thrown inside release, or improper async cancellation timing), leaks will happen.

Every connection checkout must be wrapped in async with or a strict try...finally. A single unhandled exception path is enough to eventually saturate the entire pool.

The simulator also injects a 1% probability of transient database errors (simulating connections reset by the peer) to verify that error classification and slot return logic are sound.

Impact of Async Cancellation and Exception Propagation on finally

In asyncio, Task.cancel() injects CancelledError into the coroutine. Its propagation path is similar to normal exceptions and still triggers finally. But the danger is: if the coroutine executes await inside finally (e.g., asynchronously closing a connection) and the cancellation signal arrives again at that moment, finally may be interrupted, leaving release logic half-done. Additionally, improper use of asyncio.shield, or catching and swallowing CancelledError in finally, can break the cleanup chain.

Therefore, "having finally means safe" is an illusion. The robust approach is to make the release path as synchronous as possible with no await, or use async with context managers to bind acquisition and release as an atomic pair, with the driver layer enforcing reclamation after checkout timeout to prevent a single coroutine's exception timing from permanently occupying a slot.

Type-Safe Configuration and Report Serialization

Benchmarks must not silently fail due to invalid input. The original uses Pydantic v2 to define LoadTestConfig, constraining pool size, concurrency, timeout, leak probability, and task duration via Field(gt=0) and Field(ge=0.0, le=1.0). Any negative or out-of-range parameter is rejected before dispatching async tasks.

The report model TestReport aggregates total requests, successes, failures, timeouts, detected leaked connections, average and max wait times, and retains per-request details. The serialization layer prefers orjson (with OPT_INDENT_2 formatted output) and falls back to standard json when unavailable, ensuring the tool runs with minimal dependencies.

Default parameters: pool size 10, concurrency 30, request timeout 0.5 seconds, leak probability 10%, task duration 0.1 seconds. This configuration is enough to expose queueing delay and capacity degradation within seconds.

Pydantic v2 Validation Timing and Performance Characteristics

Pydantic v2 uses pydantic-core, written in Rust, for validation. Type conversion and constraint checks complete at model instantiation; failures raise ValidationError aggregating all field errors. Constraints like Field(gt=0) and Field(ge=0.0, le=1.0) are compiled into validators at model definition time, with runtime overhead far lower than v1's pure Python implementation.

For benchmark tools, this means invalid configurations (e.g., negative pool size, leak probability > 1) are rejected before any async task dispatch, avoiding the embarrassment of "crashing halfway through." Combined with model_dump(), it directly produces a serializable dict, then handed to orjson or standard json for output—forming a clean boundary of "validate—execute—report."

Three Actionable Tuning Principles

Based on simulation results, the original distills three principles worth taping to every backend engineer's monitor:

1. Pool size is a constraint, not a buffer. Blindly increasing pool capacity only moves the failure point from the application layer to the database kernel, potentially causing max_connections saturation, context-switch overhead, and OOM.
2. Use acquisition timeouts for fast failure. Unbounded pool acquisition waits turn transient traffic spikes into cascading worker starvation. Always set explicit timeouts at the checkout layer.
3. Audit async cleanup contexts. Every connection checkout must have a paired release path. An event loop task that leaks a slot is a time bomb in a long-running service.

Conclusion

Connection pool tuning requires observable experiments, not guesswork. With a simulator built on asyncio.Semaphore, teams can safely stress-test pool exhaustion thresholds in CI environments, quantify queueing delay distributions, and verify that leak protections actually work. The complete code from the original article is available on Gumroad as "pay what you want," suitable for integration into existing failure drill processes.

Rather than bumping up connection counts at 3 AM in response to timeout alerts, simulate the problem clearly in advance with a few dozen lines of code.