Before Choosing an LLM API Reseller: A 5-Metric Self-Evaluation Checklist (With Python Script)

When you buy API access to the same model through different resellers (relay providers), you quickly discover they're not the same thing at all. Some time out constantly during peak hours, some have a decent median latency but terrible P95 performance, and some even quietly slip error messages into response bodies that return HTTP 200.

If you're choosing an API reseller for the same model, making decisions based on gut feel often leads to trouble. The checklist below covers 5 key metrics you should verify before committing, along with a Python probe script you can run yourself.

Why "Same Model" Doesn't Mean "Same Service"

Different resellers may sell access to the exact same model name, but their differences in network routing, load balancing, and rate-limiting policies will directly determine your real-world experience. The core tension is this: model quality is one thing, service reliability is another. Conflating the two is the most common mistake.

To make a sound choice, the key is breaking evaluation down into quantifiable, reproducible dimensions — not relying on subjective impressions like "this one feels fast."

The Five Core Metrics

1. Compare Apples to Apples

Only compare resellers offering the same model name. The moment you mix in different models, the conversation shifts from "reliability" to "quality," and your conclusions become distorted. Make sure you're testing the same model field across providers — that's how you control the variable.

2. Measure Availability — Don't Guess

Any request that results in a timeout, a non-2xx status code, or an error payload should be counted as a failure.

More importantly, always report success rates alongside sample size. "19/20" and "950/1000" represent completely different levels of evidence — the former could just be luck, while the latter has statistical weight. Don't let a small sample give you false confidence.

3. Watch P95, Not Averages

What users actually feel is the "long tail" of slow requests — and averages are precisely what hides that tail. A service with a beautiful median latency might have P95 performance that makes some requests unbearably slow. When measuring latency, P95 (the 95th percentile) reflects the true worst-case experience range far better than the mean.

4. Timestamp Every Measurement

A reseller that performs well at 03:00 AM might completely fall apart at 21:00. Network conditions and load fluctuate over time, so every data point must carry a timestamp. Performance data without temporal context has very limited reference value.

5. Mark Missing Data as Missing

If you weren't able to test a particular provider, clearly mark it as "no data" — never fill the gap with guesswork. Passing estimates off as real measurements undermines the credibility of your entire evaluation.

A Minimal Probe Script

The Python script below implements the core ideas above: it fires N requests at a specified endpoint, calculates success rate and P95 latency, and records the time of measurement.

import time, statistics, requests, datetime as dt

def probe(base_url, key, model, n=20, timeout=60):
    lat, ok = [], 0
    for _ in range(n):
        t0 = time.perf_counter()
        try:
            r = requests.post(f"{base_url}/v1/chat/completions",
                headers={"Authorization": f"Bearer {key}"},
                json={"model": model, "max_tokens": 16,
                      "messages": [{"role": "user", "content": "Reply with OK"}]},
                timeout=timeout)
            body = r.json()
            if r.ok and "choices" in body:
                ok += 1
                lat.append(time.perf_counter() - t0)
        except Exception:
            pass
    p95 = statistics.quantiles(lat, n=20)[18] if len(lat) >= 2 else None
    return {"checked_at": dt.datetime.utcnow().isoformat() + "Z",
            "success": f"{ok}/{n}", "p95_s": p95}

A few design details worth noting: the script uses r.ok and "choices" in body as a dual success check — verifying both the status code and the response body structure, which directly addresses the "error hidden inside a 200" problem from metric #2. It uses statistics.quantiles to approximate P95 at the 18th quantile point, echoing metric #3. And the return value includes a checked_at timestamp, fulfilling metric #4.

One important reminder: only probe endpoints you are authorized to test, and keep n at a modest value to avoid hammering any provider with stress-test-level traffic. This is both a matter of technical ethics and a practical concern — large request volumes can trigger rate limiting that skews your results.

---

The line statistics.quantiles(lat, n=20)[18] is worth unpacking. Python's standard library quantiles function divides data into n equal parts and returns n-1 cut points. With n=20, it returns 19 quantile points: index 0 corresponds to the 5th percentile (P5), and index 18 corresponds to the 95th percentile (P95). This is a lightweight way to approximate P95 using the standard library, with no need for external dependencies like numpy. Note that when the sample size lat is small (e.g., only 5–10 successful requests), the quantile estimate carries significant error — which is why the script only calculates it when len(lat) >= 2 and why using a reasonable value of n (20 or more) is recommended. In scenarios requiring higher precision, you can switch to numpy.percentile(lat, 95) or simply increase the number of samples.

Don't Want to Run the Script Yourself?

If you'd rather not build this pipeline from scratch, the original author mentioned using Folkbench for exactly this purpose: it ranks resellers for the same model against a consistent set of public metrics (availability, response time, pricing), timestamps every result, keeps missing data visible, and separates paid placements from ranking order.

Note that this is a single-source recommendation from the original author. Before relying on it, verify that its metric definitions align with your own requirements.

Summary

Choosing an LLM API reseller is fundamentally a reliability evaluation, not a model benchmark. Put these five principles into practice — apples-to-apples comparison, measured availability, P95 over averages, timestamped data, and missing data stays missing — and pair them with a lightweight script for real-world testing. That turns "which provider is better" from guesswork into a data-backed judgment. What additional metrics you track beyond these — billing accuracy, long-connection stability — is exactly the kind of experience worth building over time.

Background Notes

P95 (the 95th percentile) is a statistical concept: if you sort all request latencies from lowest to highest, P95 is the value below which 95% of requests fall — in other words, it marks the entry threshold for the slowest 5% of requests. By contrast, the average gets pulled up by a handful of extremely slow requests while simultaneously being pulled down by the large volume of fast ones, resulting in a number that looks acceptable but masks real pain points. In practice, if P95 latency reaches 10 seconds, it means roughly 1 in every 20 users will experience a frustrating wait — which causes significant user drop-off in high-concurrency scenarios. For stricter evaluations, P99 or even P99.9 may also be tracked to capture extreme long-tail behavior, depending on how much tolerance your business has for occasional slowdowns.

"Error messages hidden inside HTTP 200 responses" is a particularly insidious failure mode in LLM API relay services. Standard HTTP protocol dictates that 2xx means success — but some resellers, when faced with upstream rate limiting, exhausted balance, or model unavailability, still return HTTP 200 while embedding the error in a JSON field (e.g., {"error": "upstream overloaded"}), rather than using the standard choices structure. If your client only checks the status code without parsing the response body, these failures get miscounted as successes, inflating your availability numbers. This is exactly why the script uses r.ok and "choices" in body as a dual check — the former validates the HTTP layer, the latter validates the business logic layer, and neither alone is sufficient.