Model Aliases Are Not Versions: Monitoring Free API Changes with a Drift Canary

A Recurring Misconception

"Nothing changed on my end." — This sentence shows up again and again in countless evaluation regression threads. The model string in the config file is identical, the prompt file is untouched, so the blame falls on randomness.

The problem is not that simple. The developer community has long circulated five seemingly reasonable but costly assumptions: pinning a model name means pinning the results; temperature=0 guarantees the same input produces the same output; an answer that looks fine means the model hasn't changed; one good demo proves the endpoint is healthy; a free server can run these checks unattended forever.

Each of these assumptions passes the "sounds reasonable" test, but each one can also cost you a week of debugging time.

Aliases vs. Snapshots: What Exactly Are You Pinning?

A model name is a pointer, not a checksum. This is the core argument of the entire article.

Provider documentation typically distinguishes two separate concepts: an alias is a moving label pointing to the current snapshot, which the provider actively rotates; a dated snapshot carries a date suffix, is usually frozen, but still sits within a deprecation window. Most projects pin an alias and then call it a "version."

The correct mental model is: an alias is like a DNS record, a dated snapshot is like an IP address. Both move, but one moves far more frequently.

Each run should record four things: the requested model string, the model string echoed in the response (if any), a UTC timestamp, and the endpoint base URL. The last item is often overlooked — the same alias may route to different backends over time.

Model alias rotation is a normal operational practice for LLM API providers: to let users automatically receive performance improvements or cost optimizations, providers point the same alias (e.g., gpt-4o) at a new snapshot, with an advance notice window documented (usually several weeks). Rotation may be triggered by model updates, hardware migrations, security fixes, or routing policy adjustments. For developers, this means "pinning an alias" only guarantees the call path is stable, not that the underlying weights, tokenizer version, chat template, or inference backend are stable. Dated snapshots (e.g., gpt-4o-2024-08-06) are frozen but still have a deprecation cycle; once deprecated, the alias keeps moving while the old snapshot may simply become unavailable. Production systems should therefore record both the alias and the snapshot identifier echoed in the response, and track "the snapshot the alias resolves to" as an explicit variable in the evaluation pipeline.

Temperature 0 Does Not Equal Reproducibility

The question nobody asks: does temperature=0 actually turn off sampling? No. It just turns a knob.

Other sources of variance remain: batch inference groups your request differently on each call; floating-point accumulation order is not fixed; mixture-of-experts routing and speculative decoding introduce jitter; a single alias may be served by multiple backend pools.

The verification method is simple: loop the same prompt twenty times and count how many distinct outputs you get. If the number is greater than one, your exact-match assertions are already broken. The correct model is — model version is a distribution, not a binary state. You compare distributions, not individual strings.

Mixture-of-experts routing (MoE routing) is a structure used by many large models: only some expert networks are activated per token or per layer, with the router dynamically selecting experts based on input. Even at temperature=0, the router's top-k selection may differ slightly due to batch composition, numerical precision, or implementation details, causing different experts to be activated and thus changing the output. Speculative decoding uses a small model to draft multiple tokens, which a large model then verifies in parallel; the verification process's dependence on other requests in the batch, accept/reject thresholds, and hardware floating-point behavior can all introduce non-determinism. These two mechanisms are major sources of irreproducibility at temperature=0, and they explain why looping the same prompt twenty times may yield multiple distinct outputs.

Eyeballing Is the Worst Detector

When a snapshot changes, the output usually reads better, not worse. It gets shorter, friendlier, more structured. Nothing seems wrong — which is precisely the problem.

The alternative is to measure cheap metrics: self-consistency (agreement frequency across K samples of the same prompt), output length distribution (a stable model keeps a stable shape), character n-gram overlap (a crude but effective fingerprint comparison), and error/refusal rates (tracked separately, never mixed into the drift signal).

These metrics cannot prove the weights are identical, but together they catch the replacements that actually hurt you.

The Two-Command Drift Canary: Full Implementation

The original article provides a detection tool that relies only on the standard library and targets OpenAI-compatible endpoints, labeled as a "proposed approach, not executed in the author's environment."

Prompt set design: keep it small and boring. Twelve prompts is a reasonable starting point, and constrained tasks quickly expose format changes. Examples include "Reply with exactly OK," "List the three primary colors, lowercase, comma-separated," and "What is 17 * 23? Reply with just the number."

Record and compare logic: record mode samples each prompt K times (default 5), recording the success count n_ok, self-consistency score, mean length, and set of character trigrams; compare mode computes the Jaccard similarity and length change rate between baseline and session, flagging drift when similarity falls below 0.90 or length changes by more than 15%.

Core structure of the full script:

**Jaccard similarity** measures how similar two sets are, defined as the intersection size divided by the union size, ranging from 0 to 1. In drift detection, model output is typically split into a set of character trigrams, then the Jaccard coefficient is computed between the baseline set and the current set. It is sensitive to word order and local phrasing changes, but does not directly penalize overall length differences, so it is often used together with the length change rate. A Jaccard below 0.90 is a heuristic threshold: it means the trigram overlap between two outputs is less than ninety percent, possibly corresponding to a snapshot replacement, template change, or decoding behavior change. The metric is crude but cheap, suitable as the first sieve in a smoke test, and cannot replace human review or full regression evaluation.
# canary.py key fragment
BASE = os.environ["LLM_BASE_URL"].rstrip("/")
MODEL = os.environ["LLM_MODEL"]
K = int(os.environ.get("CANARY_SAMPLES", "5"))

def call(prompt):
    body = json.dumps({"model": MODEL, "temperature": 0,
                       "messages": [{"role": "user", "content": prompt}]}).encode()
    req = urllib.request.Request(BASE + "/chat/completions", data=body,
        headers={"Content-Type": "application/json", "Authorization": "Bearer " + KEY})
    with urllib.request.urlopen(req, timeout=60) as r:
        d = json.load(r)
    return d["choices"][0]["message"]["content"] or ""

Thresholds need to be tuned to your own noise floor; 0.90 and 15% are just starting guesses, not laws.

Scheduling on a Free Server

The canary is an extremely lightweight workload — a few short requests per cycle is enough, which makes free servers a legitimate scheduling host.

The original article discloses here: this article was prepared as part of a MonkeyCode product promotion, and the platform offers free model access and free server options, covering both ends of this loop.

Pace it according to your own quota: 12 prompts × 5 samples every 30 minutes accumulates quickly; 3 samples per hour is a calmer default. The cron configuration needs flock -n to prevent overlapping runs.

Several key limitations of free servers: idle sleep breaks the hourly promise, and gaps are not evidence of stability; ephemeral disk loses data on restart, so results must be pushed to external storage each cycle; egress and quota limits may truncate a run midway, so record n_ok to notice; no SLA means the monitor's own uptime is also a variable.

cron and flock are a common combination for scheduling a canary on a free server. cron triggers the script on a schedule but cannot prevent the next run from being launched while the previous one is still running; if a single run times out due to network retries or quota waits, overlapping runs will pollute the baseline and waste quota. flock -n attempts to acquire an exclusive lock on a specified lock file, exiting immediately if the lock is already held, thus guaranteeing only one canary process runs at a time. Combined with external storage (such as object storage or another free service) to save each cycle's results, it mitigates the problem of free-server ephemeral disks losing data on restart. Note that idle sleep may prevent cron from triggering at all, so the monitor's own uptime needs to be observed separately.

Decision Table for Interpreting Output

Signal	Likely Cause	Action
n_ok below K for many prompts	Auth, quota, or network issue	Fix access first; this is not drift
self_consistency drops sharply	Decoding or routing change	Freeze the release, rebuild the baseline
Jaccard below 0.90 on constrained prompts	Snapshot replacement	Pin the dated snapshot, rerun evaluations
mean_len up more than 15%	Chat template change	Check prompt wrapping and stop reasons
Everything stable	Nothing proven	Keep the baseline, mark the date

The last row is the most important: a clean canary run proves nothing — it only means you didn't catch a change this hour.

The Boundaries of This Workflow

It cannot prove the weights are identical, only observe behavior; twelve prompts are a smoke test, not a regression suite; thresholds are guesses until you measure your own variance; if you never separate errors from drift, errors will masquerade as drift; baselines silently rot, so you must recollect after planned upgrades.

When not to use it: early prototypes where model replacement is costless; teams with contracts guaranteeing changelogs and notice windows; people who cannot store baselines outside the free box; those who need reproducibility guarantees rather than a detector.

The Corrected Mental Model

1. A model name is a pointer, and pointers move.
2. Temperature 0 narrows the distribution, never collapses it.
3. Compare distributions, not individual outputs.
4. Errors and drift are different signals; label them separately.
5. Every baseline has an expiration date; recollect deliberately.

The original article ends with a question: which twelve prompts would you put in your set? Choose the tasks where a silent change hurts the most, write them down today, and let the canary tell you when the pointer moves.

It should be noted that the core arguments of this article — aliases are not versions, temperature 0 is not reproducible, eyeballing is unreliable — are widely accepted engineering consensus in current LLM API practice; but the specific drift thresholds and free-server scenario advice are labeled in the original as unexecuted proposals, and readers should treat them as starting points rather than validated conclusions.