LLM Performance Drift: 31,352 Measurements Reveal the Truth About API Model Stability

When Benchmarks Shift from Snapshots to Time Series

For a long time, our understanding of large language model (LLM) benchmarks has rested on an implicit assumption: a model gets evaluated, a score gets published, and then we discuss that score as if it describes a relatively stable object. But for models served via API, what lies behind a "model name" can change over time — the serving infrastructure changes, provider configurations change, versions change, and sometimes behavior shifts quietly even without any obvious public version update. Major LLM API providers (OpenAI, Anthropic, Google, etc.) vary significantly in their model versioning practices. OpenAI uses date-stamped snapshot names (e.g., gpt-4-0613) and commits to consistent behavior for specific snapshots over a defined period, but the general-purpose aliases (e.g., gpt-4) silently switch the actual version they point to. Anthropic identifies versions through date suffixes in model IDs. However, even within the same named version, providers may apply inference optimizations (such as quantization, speculative decoding, or KV cache strategy adjustments) or infrastructure changes (such as switching GPU models or adjusting batching strategies) — changes that aren't reflected in the version number but can subtly shift the model's output distribution. This "nominally the same, substantively different" situation is precisely the core problem that longitudinal evaluation aims to capture.

The founder of the AI Stupid Level platform shared a thought-provoking observation on Reddit: they've stopped treating benchmarks as a "leaderboard competition" and instead view them as a longitudinal measurement problem. Longitudinal measurement is a classic methodological concept from the social sciences and medical research, referring to repeated measurements of the same subject at multiple time points to capture its trajectory of change over time. Unlike a cross-sectional study that takes a single snapshot, longitudinal research can distinguish "within-subject change" from "between-subject differences." Applying this framework to LLM evaluation means we no longer treat each benchmark run as an independent event, but instead view multiple scores for the same model as a time series — enabling the application of statistical tools like trend analysis, seasonal decomposition, and change point detection. Their team continuously evaluates model performance on coding, multi-turn reasoning, and tool use, while running lightweight probes at higher frequency.

This shift in perspective means the core question is no longer "which model scores highest," but rather a series of more nuanced diagnostic questions: Has the model's behavior drifted from its own prior baseline? Does this change exceed its normal range of repeated-call variability? Has the benchmark configuration itself changed? Is the effect concentrated on a specific task? Is there correlation across multiple models from the same provider? And — could what appears to be a capability degradation actually just be an availability or infrastructure issue?

Core Data: What the 3:1 Variance Ratio Reveals

The central finding of this analysis comes from a remarkably solid dataset: 31,352 repeated scoring observations across 49 models.

The results yielded two key numbers:

• Within-day score standard deviation: 2.80 points
• Between-day daily median standard deviation: 8.43 points

That's roughly a 3:1 ratio.

The distinction between within-day and between-day variance belongs to the classic framework of variance decomposition in statistics, commonly seen in repeated-measures ANOVA and ICC (intraclass correlation coefficient) calculations. Within-day variance reflects the random noise of the measurement instrument itself — in the LLM context, this primarily comes from sampling temperature during inference, random seed differences across API calls, and similar factors. Between-day variance, on the other hand, may contain signals of systematic change, such as provider model weight updates, inference infrastructure configuration changes, or load-balancing strategy adjustments. A 3:1 variance ratio means that systematic day-to-day fluctuations far exceed pure measurement noise — in signal detection theory, this is a signal strength worth paying attention to.

In other words, the variability produced by calling the same model multiple times within a single day is far smaller than the variability observed across different days.

The author was notably restrained in pointing out that this result alone does not prove that providers are modifying models on a daily basis — because too many potential confounding factors exist: task composition, sampling, missing data, provider behavior, and methodological changes could all contribute to this difference. But it's enough to establish one thing: variation along the time dimension deserves dedicated measurement and should not be dismissed as noise around some "permanent leaderboard score."

This is an important methodological stance. In most evaluation reports, model performance is treated as static by default. This work reminds us that the model behind an API is a "moving target," and any single-point measurement may be nothing more than an incidental result from a particular time slice.

A Rigorous Longitudinal Evaluation Methodology

Based on these insights, the team adopted a carefully designed measurement methodology with several features worth borrowing for engineers working in production ML environments:

Versioning and Compatibility Comparisons

They apply version control to benchmark configurations, only comparing longitudinal observations under compatible measurement conditions. This avoids apples-to-oranges comparisons — if the benchmark configuration itself has changed, then score changes cannot be attributed to the model.

Execution-Based Evaluation, Not LLM Judges

Wherever possible, they use execution-based repeated evaluation rather than using another LLM as a "judge." This point is particularly critical.

LLM-as-a-Judge — using a large language model to serve as an evaluator — is a popular approach in recent LLM evaluation, originally popularized by LMSYS in their Chatbot Arena and later adopted by multiple benchmark platforms. The core idea is to have strong models like GPT-4 score or pairwise-compare outputs from other models. However, this approach has multiple known flaws: first, position bias, where the judge model tends to favor responses in certain positions; second, self-preference bias, where models tend to score outputs similar to their own style more highly; and most critically, the judge model itself is also called via API and equally subject to version drift. This creates a measurement nightmare: your measurement instrument and the object being measured may both be undergoing uncontrolled changes simultaneously, making it impossible to attribute observed score changes. Using LLM-as-a-judge to measure drift is like measuring length with an elastic ruler.

Separating Availability Failures from Task Results

They separate availability failures from valid task results. This directly addresses the diagnostic question raised earlier — a timeout or service outage should not be misinterpreted as model capability degradation. Additionally, when providers expose relevant information, they track service and version metadata and run change point detection on the resulting time series.

Change point detection is a core problem in time series analysis, aimed at identifying moments when the statistical properties (such as mean, variance, or distribution) of a data-generating process undergo abrupt shifts. Classic algorithms include CUSUM (Cumulative Sum Control Charts), PELT (Pruned Exact Linear Time), and Bayesian Online Change Point Detection. In the context of LLM performance monitoring, change point detection can help automatically identify sudden shifts in model behavior — for example, a model's code generation capability suddenly dropping 5 points starting on a certain day, potentially corresponding to a silent provider update. However, the unique challenges of LLM performance time series include: relatively sparse data points, high noise levels, and the possible presence of changes at multiple scales (gradual capability decay vs. abrupt version switches), which may lead to elevated false positive rates for traditional change point detection algorithms.

Benchmark Contamination: More Transparency, More Distortion

This work also touches on an increasingly pressing issue: benchmark identification and contamination.

Benchmark contamination refers to evaluation data leaking into a model's training data in some way, artificially inflating the model's benchmark performance rather than reflecting genuine capability improvement. This problem was already a concern with traditional NLP benchmarks (such as SQuAD and GLUE) but has become especially severe in the LLM era. Contamination can occur through multiple pathways: benchmark questions published on public web pages get collected by crawlers into pretraining corpora; the open-source community exposes test sets when reproducing evaluation code on GitHub; and providers may even indirectly optimize for known benchmarks through data selection during the RLHF (Reinforcement Learning from Human Feedback) stage. Recent research shows that some models score 10–15 percentage points higher on benchmark subsets known to be contaminated compared to uncontaminated subsets, seriously undermining the credibility of benchmark scores as proxy indicators of capability.

Once a benchmark becomes well-known enough, publicly releasing every live task, prompt variation, and hidden test can itself alter what you're trying to measure — this content may enter training data or be specifically optimized against by providers.

There is a fundamental tension here: science demands transparency and reproducibility, but transparency itself can undermine measurement validity.

The team's strategy is to separate methodological transparency from publishing the complete live evaluation set. They've made the full methodology publicly available (a public PDF) detailing the measurement design, assumptions, limitations, and statistical interpretation, while keeping the exact live task pool and certain operational parameters private.

This is a pragmatic compromise: making the methodology open to scientific scrutiny while not handing the "exam questions" directly to the test subjects.

Open Questions for the LLM Evaluation Community

Adopting a researcher's posture, the author poses several genuine technical questions to peers working in evaluation, change point detection, and production ML:

• For longitudinal LLM evaluation, should daily medians serve as the primary time series unit, or should individual repeated observations be modeled directly?
• When version metadata is incomplete, how can genuine model drift be distinguished from provider/infrastructure effects?
• How much should a live benchmark conceal to reduce contamination while still maintaining methodological scientific verifiability?
• For these non-stationary, relatively noisy model performance series, are there methods better suited than change point detectors?

These questions have no standard answers, but they point toward a direction in LLM evaluation that has not yet received sufficient attention.

Conclusion: Treating "Drift" as a First-Class Citizen

As more and more applications depend directly on API-served models, model behavior stability is no longer just an academic curiosity — it's a real engineering risk. A prompt that works perfectly today might degrade tomorrow due to silent server-side changes.

The greatest value of this work, based on 31,352 measurements, may not lie in the 3:1 variance ratio itself, but in the framework it proposes: treat temporal variation as a first-class citizen that demands serious measurement, not as background noise around a leaderboard score.

For any team relying on LLMs in production, this is a reminder worth taking seriously — the model you depend on may not be as stable as its name suggests.

Disclosure: The original poster disclosed that they are the founder of the AI Stupid Level platform, and the purpose of the post was to solicit technical criticism of the methodology, not to promote a commercial product. Public methodology PDF: aistupidlevel.info