Stop Trusting 99% Uptime: How to Actually Test LLM API Relay Stability

Relay services that claim "99.9% uptime" and advertise "dedicated high-concurrency channels" often turn out to be a different story once you connect. The Folkbench team turned a production incident into an automatable acceptance-testing methodology for relays, and the core conclusion is simple: status-code monitoring gets completely fooled by fake 200s, silent truncation, and late-night time-to-first-token delays.

Why Traditional Health Checks Fail

Fake 200s and Silent Truncation

Their initial detection method was naive: a Python script sent a POST to the relay every 5 minutes, and if it got an HTTP 200 with text, it was deemed healthy. Within two days of launch, users reported that the model stopped mid-sentence. Digging through logs revealed that when the upstream pool was exhausted (429/401), the gateway timed out (504), or a node crashed, some relays still returned 200 OK.

Even more insidious is how they handle the middle of a stream: some quietly insert data: [DONE] to end it, while others stuff {"error": "rate limit exceeded"} into the JSON, disguising it as the model's own answer. To monitoring that only looks at status codes, all of this is "success."

SSE (Server-Sent Events) is the underlying protocol for streaming responses from LLM APIs like OpenAI. Built on HTTP long connections, the server pushes events one by one in the format data: <payload>\n\n, ending with data: [DONE]. The key point is that SSE itself has no enforced termination semantics—the connection is "soft-closed," and the client can only determine the end by receiving [DONE] or a socket disconnect. This means middleboxes (proxies, gateways, relay layers) can legitimately truncate the stream at any point, and if the client only validates the HTTP status code, what it receives is still a "200 success." This is the technical soil in which fake 200s and silent truncation take root: the status code describes connection establishment, not response completeness.

Nighttime Stalls: First-Token Latency Hidden by Averages

During the day, average response time was stable at around 2 seconds. During nighttime peaks, the rate of timeouts and voluntary abandonments began to climb. The worst case was when the entire answer still completed in about 4 seconds, but time to first token reached 12 seconds—users staring at a blank box for a few seconds will leave.

A single ping, or casually firing off two requests, doesn't even begin to count as testing a streaming AI API.

Three Generations of Test System Evolution

Folkbench summarized their probe system iterations into three stages:

• 1.0 Blind Testing: Manually trying a few prompts in the Web UI, or hand-writing 10 requests. Small sample size, high noise, and occasional disconnections only surface after launch.
• 2.0 Concurrency + Status Codes: Load testing with Postman / Locust, tracking status code distribution, average latency, and success rate. You can see the outer gateway, but not broken streams, forged errors, or pauses between tokens.
• 3.0 Stream State Machine: Treating the relay as a long-connection network middlebox (similar to real-time audio/video or market data feeds, not ordinary web requests). Multi-region probes separate connection establishment, TLS handshake, first token arrival, token throughput, and SSE parsing into distinct measurements.

This shift in analogy is the most valuable part of the entire article: a relay's failure modes are not the same as those of an ordinary HTTP service.

Three Dimensions That Determine Production Experience

Latency: Separating "Waiting" from "Generating"

• TTFT (time to first token): From request sent to first SSE packet received. Their production standard is TTFT < 1.5 seconds.
• TPS: Number of generated tokens after the first token divided by elapsed time.
• P95 / P99: Good averages are meaningless.

TTFT (Time To First Token) is measured separately because streaming LLM latency consists of two fundamentally different phases: the "cold start" phase of queuing + first-token inference, and the "steady state" phase of subsequent token-by-token generation. The former is determined by upstream pool scheduling, concurrent queuing, and prompt pre-filling; the latter depends on decoding throughput. Blending them into a single average response time allows two types of failures to cancel each other out—for example, if the first token takes 12 seconds but generation is fast, the total time looks normal while the user experience has already been destroyed. In production, therefore, SLOs must be set separately: TTFT reflects interactivity, TPS reflects sustained output capability, and degradation in either cannot be masked by the other.

Reliability: Seeing Through Fake Success

• Parse success rate: The official SDK must be able to parse the payload, the JSON must be complete, and the answer must not contain relay-injected errors.
• Retry-to-success: With exponential backoff, can 429/5xx recover within 3 retries?

Parse success rate and HTTP success rate are two different things. The former requires the client to actually consume the payload using the official SDK or a strict JSON parser: SSE chunk boundaries must be correct (a single JSON object cannot span multiple data: events), the JSON must be fully closed, and no forged content injected by the relay layer may appear. Many relays interleave {"error": ...} into the stream or package their own generated prompts as model answers—completely normal from a status code perspective, but exposed during parsing or manual review. Making this an independent metric is essentially using "whether downstream can actually use it" to reverse-validate "what upstream claims as success," and it is the most direct criterion for seeing through fake success.

Stream Quality: Catching Silent Failures

• Stream interruption rate: The proportion of calls that drop or go silent before data: [DONE].
• Inter-token jitter: The standard deviation of intervals between SSE packets. High jitter means an unstable path or buffering stacked at the proxy layer.

Methodological References

This methodology wasn't developed in isolation—the authors list three references:

• Google SRE Four Golden Signals: Latency, traffic, errors, saturation—treating the relay as an untrusted external service.
• OpenTelemetry + W3C Server-Sent Events: Recording TCP handshake, TLS, response headers, first chunk, and [DONE].
• Wilson Score Interval: 10/10 (100%) is not automatically more trustworthy than 9,900/10,000 (99%). They rank by confidence lower bound:

Score = (p̂ + z²/(2n) − z·√(p̂(1−p̂)/n + z²/(4n²))) / (1 + z²/n)

Where p̂ is the observed success rate, n is the sample size, and z is the normal quantile corresponding to 95% confidence. Small samples are penalized.

How to Build the Probes

• Multi-region probes (AWS US-West, GCP East Asia Hong Kong/Tokyo, Europe) to distinguish "relay is stuck" from "upstream fluctuation."
• Socket state machine: CONNECTING (t0) → FIRST_BYTE (TTFT = t1 − t0) → STREAMING (intervals, TPS, jitter) → COMPLETED ([DONE] present, JSON valid).
• Automatic attribution:
  • 502/504 → The relay's own gateway node failure.
  • 429/401 → Concurrency control failure or upstream key balance exhausted.
  • 200 + stream interruption or parse failure → The relay's billing/filtering middleware is overloaded.

Three Rules Before Choosing a Relay

1. Don't test just once. Sample continuously for at least 3–7 days across day and night, focusing on P95 and success rate during the 21:00–01:00 peak hours.
2. Always test streaming. Non-streaming masks first-token latency, buffering, and mid-stream death.
3. Suspiciously low prices are not a gift. They usually mean shared account pools, poor-quality proxies, or cheap open-source models being sold as premium ones.

Folkbench publishes P95 latency, availability, and price per token for each relay publicly, viewable without registration. For developers, this is at least more informative than a lone "99% uptime" badge.

Back to the methodology itself: relay quality is a distributed systems problem, not a marketing copy problem. When you upgrade your probes from "request-response" to "long-connection state machine," and swap your metrics from "averages" to "P95 + stream quality attribution," most forged successes will reveal themselves.

The Wilson score interval is a type of confidence interval for proportions. Compared to the naive normal approximation (Wald interval), it doesn't produce extremely narrow or even out-of-bounds intervals when the sample size is small or the success rate is near 0/1, making it better suited for conservative ranking of "success rates." The article uses its confidence lower bound for ranking, meaning: with small samples, even if 100% success is observed, the lower bound remains significantly below 100%, preventing "10 out of 10 passed" from being misjudged as better than "9,900 out of 10,000 passed." This is precisely the core requirement of vendor selection—using statistical conservatism to combat sample noise, rather than making decisions based on a single flattering observation.

One-line Summary

Relay stability must be verified with stream state machines and P95 metrics; status code monitoring gets fooled by fake success.

Paragraph Summary

The article reviews a production incident from the Folkbench team, pointing out that LLM API relays claiming "99.9% uptime" actually have hidden failures such as fake 200s, silent truncation, and late-night first-token latency, and that traditional status-code-based health checks fail completely. The team proposed three generations of test system evolution: from manual blind testing and concurrent load testing, upgrading to stream state machine probes that treat the relay as a long-connection middlebox, separately measuring TTFT, TPS, P95/P99, parse success rate, retry-to-success, stream interruption rate, and inter-token jitter. Methodologically, it references Google SRE Four Golden Signals, OpenTelemetry, and the Wilson score interval. Multi-region probes plus socket state machines and automatic attribution can distinguish relay failures from upstream fluctuations. Before choosing a vendor, sample across day and night for 3–7 days, focusing on streaming and peak-hour performance.