The Reproducibility Crisis in Hosted LLM Behavioral Evaluation: Measurement Methods Matter More Than Models
The same model with the same evaluation method can yield opposite conclusions at different times — measurement tools matter as much as models.
The arXiv paper *Replication Without Persistence in Hosted LLMs* identifies a fundamental dilemma in hosted LLM behavioral evaluation: when two evaluations disagree, it's impossible to tell whether the model service or the measurement tooling changed. Using the Regent Chess sequential decision environment and Gemini models as a case study, the paper finds that a historical deficiency replicates (endpoint +0.0530), but rebuilding the evaluation configuration alone significantly reduces the endpoint (-0.0429), and the comparison between Gemini 3.1 and 3.7 reverses sign under the rebuilt configuration. All three validation dimensions yield contradictory conclusions, requiring explicit annotation of model identifier, serving period, measurement instrument, and inference configuration for any performance claim.
Why Does the Same Model Produce Different Results on Two Evaluations?
Behavioral evaluation of hosted large language models (Hosted LLMs) is facing a thorny problem: the same model, evaluated with the same methodology, can yield completely different conclusions when run at different times. A new paper on arXiv, Replication Without Persistence in Hosted LLMs, targets this pain point directly — when evaluation results change, we often cannot determine whether the service being tested changed, the measurement tooling changed, or both changed simultaneously.
This challenge undermines trustworthy evaluation across the entire industry. Vendors continuously update their online models, and the actual service behind a public identifier (such as gemini-3.1-flash-lite) may have quietly changed. Meanwhile, researchers also iterate on their evaluation scripts and inference configurations. When two variables are entangled, any conclusion that "a model performs worse" may not hold up.
Three Validation Questions That Must Be Separated
The paper's core contribution is decomposing "validation" into three independent questions, to avoid conflating them:
Replication
Given unchanged historical configuration, re-run with fresh data and see whether prior findings reproduce. This is the most classic scientific replication concept.
Measurement Sensitivity
Keeping the same model identifier, but rebuilding the evaluation and inference configuration entirely — does the endpoint metric change? In other words, how much impact does a change in the measurement tooling itself have?
Persistence
Under a single unified tool, does a finding carry over to subsequent tests on other model identifiers? This tests the generalizability of conclusions.
The authors emphasize that these three questions can yield different answers within a single evaluation — which is precisely where current evaluation practice tends to go wrong.
This three-way distinction is rooted in discussions of the Replication Crisis in philosophy of science. Traditional scientific replication typically assumes the subject under study is stable — a chemical compound, an animal behavior. But hosted LLMs break this assumption: model services are continuously updated software systems, and vendors may silently swap underlying weights, adjust sampling strategies, or modify system prompt templates without changing the public identifier. This makes it nearly impossible to fully isolate "changes in measurement tooling" from "changes in the subject being measured" along the timeline, unless researchers have precise control over the server side. For this reason, the paper isolates "measurement sensitivity" as its own dimension to specifically quantify the noise introduced by the evaluation framework itself.
Regent Chess: A Test Environment That Aligns with Ground Truth
The study uses a sequential decision environment called Regent Chess. Its cleverness lies in the fact that it contains a hidden but variable state that is precisely recorded. This means the model's "stated beliefs" at the time of action can be scored against ground truth one-by-one.
The evaluation uses a "model-minus-uniform" endpoint metric, where positive values indicate worse performance than the baseline. This design makes abstract "belief accuracy" quantifiable and auditable, providing a clean experimental platform for distinguishing among the three validation questions.
Sequential decision environments are a common experimental setup for evaluating LLM reasoning, where models must make action choices across multiple interaction steps based on current observed state. The difficulty in most such environments is that "beliefs" are hard to quantify — the model's natural language reasoning process cannot be precisely compared against objective state. Regent Chess solves this by designing a hidden variable that is invisible to the model but precisely tracked by evaluators. The model-minus-uniform metric design is analogous to baseline correction in controlled experiments: the uniform baseline represents a lower bound for an agent with complete ignorance of the hidden state making uniform guesses, and a positive difference signals that the model performs even worse than random guessing — an intuitive signal of performance degradation.
Key Experimental Findings: Three Different Conclusions
The paper uses a previously reported performance deficiency in Gemini 3.1 Flash-Lite as its entry point and tests each dimension in turn:
Replication holds: Re-running fresh games under the historical configuration does reproduce the previously reported deficiency, with an endpoint value of +0.0530 (95% CI [+0.0329, +0.0714]). The finding is real.
Measurement sensitivity is significant: In a back-to-back H/R comparison on the same day with the same public identifier, the model-minus-uniform endpoint decreased by 0.0429 after rebuilding the configuration (H-minus-R comparison 95% CI [+0.0182, +0.0667]). The problem is that six configuration components changed simultaneously, so it is impossible to isolate which component caused the difference. Simply rebuilding the evaluation configuration is sufficient to change the conclusion.
Persistence fails: Under the rebuilt R configuration, in a pre-frozen, interleaved same-window 4K comparison, the sign reversed between Gemini 3.1 and Gemini 3.7 (two identifiers with different release times and product tiers). Additional descriptive and exploratory units showed the same directional pattern. The additional contribution of the serving period itself remained unresolved (-0.0166, [-0.0483, +0.0157]).
Implications for Industry Evaluation Practice
The core message of this research is that replication, measurement sensitivity, and persistence can yield mutually contradictory conclusions within a single evaluation. This means that any claim about hosted model behavior cannot simply state "a model is better or worse" — it must explicitly annotate four dimensions:
- The tested model identifier
- The serving period
- The measurement instrument
- The inference configuration
For developers and enterprises who rely on third-party evaluation leaderboards for decision-making, this is an important reminder: online models are not static targets. The conclusions of any single evaluation have a clear "expiration date" and applicable scope. Performance claims that lack these indexing dimensions may no longer hold on the next run.
This also sets a higher bar for the evaluation methodology community — only by decoupling server-side changes from measurement-side changes can behavioral evaluations truly achieve credibility and comparability.
The four-dimensional annotation framework proposed in the paper echoes the "Model Card" and "Datasheet" concepts advocated in the ML reproducibility field in recent years, but focuses more on evaluating behavior rather than static descriptions of the model itself. The serving period dimension in particular is critical yet often overlooked — it requires researchers to record the time window of evaluation runs so that others can judge whether cited conclusions still apply to the current service version. For API-dependent benchmarks (such as online leaderboards like LMSYS Chatbot Arena), this means that historical ranking entries are, in a strict sense, only valid for the service state within a specific time window, and cross-temporal horizontal comparisons require additional methodological safeguards before strong conclusions can be drawn.
Related articles
Why Does a 35GB Quantized Model Blow Up an 80GB GPU? The Truth About KV Cache VRAM
Why does a 35GB 4-bit quantized LLM still blow up an 80GB GPU? This article breaks down the four components of inference VRAM, provides precise KV Cache formulas, compares GQA vs DeepSeek MLA memory, and offers production-grade OOM prevention like FP8 KV Cache.
Turning Company Workflows into Agent Skills: A Practical Guide
An agency owner turns team workflows into Agent Skills: one command handles article writing, six-platform distribution, and dual human approval. This guide breaks down the Skill file structure, two invocation methods, three-tier storage layout, security review process, and how to turn your own weekly repetitive tasks into executable, auditable skills.
China Releases "Qiusuo" 4.0: The First Evaluation Benchmark Covering the Entire AI Industry Chain
China's CESI releases "Qiusuo" 4.0, the first AI full-industry-chain evaluation benchmark system, covering computing power, LLMs, agents, security governance, and industry applications with 16 evaluation tools and open-source code tools to solve cross-environment incomparability.