Can You Trust Free Models? Run an A/A Test First

Swapping a paid API for a free model, running your test suite once, and then blogging that it "works" or "doesn't work" ignores one premise: a single run of a non-deterministic system gives you a sample, not a measurement. If the free tier and the paid tier route sampling differently, comparing two samples is like comparing two coin flips and calling it experimental evidence.

A/A Testing: Measure the Noise Before You Compare

An A/B test only means something when the harness itself is quiet. The A/A test is that control group: it tells you how loud the harness is. If five identical calls return five different pieces of text, then your diff-based regression suite is measuring sampling noise and mistaking it for a regression.

Setting temperature to 0 doesn't fix this. Shared inference stacks batch and reorder work, kernels reduce in different orders, and plenty of providers quietly rotate models behind a stable alias. Determinism at temperature=0 is a common assumption, not a guarantee.

That property is also a gift in reverse: a fingerprint probe doubles as a tripwire. If an unchanged input suddenly causes consistency to collapse, something changed upstream, and it wasn't your prompt.

The Probe Script

The whole experiment needs a small script: one prompt, k repetitions, a single line of JSON output. It assumes an OpenAI-compatible /chat/completions endpoint.

The core logic: normalize each returned text (lowercase, strip punctuation, collapse whitespace), then take a SHA256 fingerprint. Count the fingerprint distribution with collections.Counter and emit modal_share (the share of the most common answer) and distinct (the number of different answers). Normalization is deliberately coarse, because what you're testing is behavior, not wording.

The SHA256 fingerprint here maps a normalized text of arbitrary length to a fixed-length hex string with negligible collision probability. For evaluation purposes, its value isn't cryptographic security but serving as an efficient proxy for "exact equality": two returned texts that are identical after normalization produce identical fingerprints; if even one character differs, the fingerprints are almost certainly different. This lets you use Counter to directly count "how many calls returned exactly the same piece of text" without character-by-character comparison or semantic similarity computation. Note that fingerprints capture only literal consistency, not semantic equivalence—two different fingerprints don't mean the answers differ semantically. That's precisely why normalization is kept as coarse as possible (lowercase, strip punctuation, collapse whitespace): erase the irrelevant formatting differences first, and what remains is more likely to reflect genuine sampling divergence.

Make It a Week-Long Record

A single probe run only tells you about one instant on shared hardware. Running it on a schedule is what adds value: you get a time series of your harness's honesty.

The script hashes the prompt file with sha256sum and writes that hash into every record. Change the prompt and the fingerprint history breaks cleanly at that version instead of quietly drifting underneath. This is the same discipline as freezing your config hash: a run has to earn the right to be compared.

This kind of task belongs somewhere other than your laptop—which is exactly where the free tier earns its keep. MonkeyCode offers free model access and a free server option, enough to host a probe loop and collect samples while you get on with other work. (Disclosure: this article is part of a MonkeyCode product promotion.)

But a shared server comes with a warning: don't run latency benchmarks against it and then report the numbers as a property of the model. You're measuring your neighbors' load.

Non-deterministic systems are computational systems that may produce different outputs for the same input. For LLM inference, sources of non-determinism include: the non-fixed order of floating-point additions in GPU parallel reductions, batch scheduling causing different requests to share different computational graph paths, and random sampling when temperature is above zero. Even at temperature=0, floating-point non-associativity produces tiny differences in logits, which can flip the argmax between close candidate tokens. A shared inference stack amplifies this further: continuous batching dynamically packs different users' requests into the same batch, and the tiling and reduction order of matrix multiplications varies with batch composition, so the same prompt may land on different computational paths at different times. Treating a single API call as a "measurement" of model behavior is therefore statistically invalid—it's just a noisy sample.

Read Your Own Numbers: A Triage Table

Here are heuristic recommendations based on modal_share (temp 0, k=5)—not measured constants. Calibrate against your own workload before citing them:

• 1.0: Supports exact-text regression diffs; does not support any semantic-equivalence judgment.
• 0.8–0.99: Supports pass@k with k≥3 and diffable error classification; does not support single-run pass/fail gates.
• 0.4–0.79: Supports aggregate scores across many prompts; does not support any per-prompt verdict.
• < 0.4: Suitable only for drafting, brainstorming, and triage; not for evaluation, CI gates, or anything auditable.

High consistency is not a quality signal. A model can be consistently wrong—that's the problem B/B testing solves, and B/B testing has to wait until A/A is quiet. Consistency is the foundation, not the score you report.

pass@k is a metric commonly used in code generation evaluation: sample the same problem k times, and the problem counts as passed if any single sample passes the unit tests. It measures "the probability that the model solves the problem within k attempts," not single-attempt accuracy. This ties directly into the A/A framework here: pass@k itself acknowledges that a single run is a random sample, which is why it requires k≥3 to be statistically meaningful. But pass@k presumes that samples are independent and stably distributed—if modal_share is only 0.5, the estimate of "at least one correct within k samples" becomes highly unstable, and comparing pass@k across runs loses its meaning. In other words, pass@k is a metric built on the premise that sampling noise has already been characterized, not a tool for characterizing that noise.

Where It Breaks, and Who Should Walk Away

Token accounting is the weak spot. If the endpoint doesn't return a usage block, the token count from your local tokenizer may disagree with the server's, and comparing budgets across two mismatched tokenizers is worse than not comparing at all. Either measure usage on both sides, or drop the token column entirely and say so.

This method also isn't for everyone. For scenarios that require pinned model versions, uptime commitments, or sub-second p99 latency, a shared free tier is the wrong tool—not because it's bad, but because those are contract problems, not experiment problems. Regulated products should be even more cautious: an evaluation system whose variance hasn't been characterized is a liability with a dashboard.

The takeaway worth keeping is narrower than "try the free tier": a free model with a known variance profile is more useful than a paid model you've never probed, because only the former lets you distinguish a regression from a re-roll. Whichever side you ultimately pick, run the probe for a week first—the consistency rate you get is the real starting line.