Folkbench
A team focused on methods for stability testing and acceptance of LLM API relay services, which proposed an evolution plan spanning three generations of probe systems, from blind testing and concurrency status codes to a streaming state machine.
Timeline (last 90 days)
Folkbench publishes P95 latency, availability and per-token price by relay station, viewable without registration.
Some LLM API relay stations still return HTTP 200 OK when the upstream pool is exhausted (429/401), the gateway times out (504) or a node crashes, causing monitoring that only looks at status codes to misjudge it as success.
The Folkbench team's production standard for streaming LLM API relay stations is TTFT < 1.5 seconds, and P95/P99 rather than averages must be monitored.
The relay station probe system proposed by Folkbench has undergone three generations of evolution: 1.0 blind testing (manually trying prompts or hand-writing a small number of requests), 2.0 concurrency + status codes (load testing with Postman/Locust), and 3.0 streaming state machine (treating the relay station as a long-connection network middlebox, with multi-region probes separately measuring connection establishment, TLS handshake, first-token arrival, token throughput and SSE parsing).
The Wilson score confidence lower bound formula used by Folkbench is: Score = (p̂ + z²/(2n) − z·√(p̂(1−p̂)/n + z²/(4n²))) / (1 + z²/n), where p̂ is the observed success rate, n is the sample size, and z is the normal quantile corresponding to 95% confidence.
Folkbench's relay station probes are deployed across multiple regions (AWS US West, GCP East Asia Hong Kong/Tokyo, Europe), used to distinguish relay station freezes from upstream fluctuations.
Folkbench recommends that before choosing a relay station you should not test only once, but sample continuously for 3–7 days across day and night, focusing on P95 and success rate during the 21:00–01:00 peak period.
Folkbench ranks relay providers for the same model by uniform, publicly disclosed metrics (availability, response time, price), annotates the detection time, keeps missing data visible, and separates commercial placements from ranking order.
All Facts (8)
Folkbench publishes P95 latency, availability and per-token price by relay station, viewable without registration.
50%UnverifiedSome LLM API relay stations still return HTTP 200 OK when the upstream pool is exhausted (429/401), the gateway times out (504) or a node crashes, causing monitoring that only looks at status codes to misjudge it as success.
50%UnverifiedThe Folkbench team's production standard for streaming LLM API relay stations is TTFT < 1.5 seconds, and P95/P99 rather than averages must be monitored.
50%UnverifiedThe relay station probe system proposed by Folkbench has undergone three generations of evolution: 1.0 blind testing (manually trying prompts or hand-writing a small number of requests), 2.0 concurrency + status codes (load testing with Postman/Locust), and 3.0 streaming state machine (treating the relay station as a long-connection network middlebox, with multi-region probes separately measuring connection establishment, TLS handshake, first-token arrival, token throughput and SSE parsing).
50%UnverifiedThe Wilson score confidence lower bound formula used by Folkbench is: Score = (p̂ + z²/(2n) − z·√(p̂(1−p̂)/n + z²/(4n²))) / (1 + z²/n), where p̂ is the observed success rate, n is the sample size, and z is the normal quantile corresponding to 95% confidence.
50%UnverifiedFolkbench's relay station probes are deployed across multiple regions (AWS US West, GCP East Asia Hong Kong/Tokyo, Europe), used to distinguish relay station freezes from upstream fluctuations.
50%UnverifiedFolkbench recommends that before choosing a relay station you should not test only once, but sample continuously for 3–7 days across day and night, focusing on P95 and success rate during the 21:00–01:00 peak period.
50%UnverifiedFolkbench ranks relay providers for the same model by uniform, publicly disclosed metrics (availability, response time, price), annotates the detection time, keeps missing data visible, and separates commercial placements from ranking order.
50%