llm-retry

In a test with 100 concurrent calls, a token bucket capacity of 20, and 10 tokens replenished per second, llm-retry's default configuration (3 retries, exponential backoff, ±10% jitter) caused 64 calls to fail completely.

In the same test, under the default configuration the 36 successful requests caused the server to additionally reject 255 requests, a request amplification factor of 2.55x.

llm-retry is a single-header C++ library that provides with_retry, with_failover, and CircuitBreaker functionality.

After raising llm-retry's max_attempts to 8, all 100 calls passed, but it took 16.6 seconds, more than double the theoretical lower bound, and sent 375 requests.

The ±10% jitter causes all clients that failed at 0 seconds to wait 450-550 milliseconds, so that 80 retries arrive crowded into the same 100-millisecond window, forming waves and failing to effectively spread out retry timing.

llm-retry's LLMError type has no retry-after field.

llm-retry's default retry_on_http_codes list does not include 529 (overloaded_error) and 408 (request timeout), causing these errors to never be retried.

llm-retry's with_failover feature silently retries 400 errors (such as prompt too long or JSON format errors) to the fallback model, masking the error and causing quality degradation that no one notices.