LLM Eval Scores Change on a Rerun — Is Your Scoring System Still Reliable?

Every time you publish AI evaluation scores, have you ever wondered: with the same input, the same model, and the same scoring rubric, will two runs produce identical results? A real-world test from rss delivered an unsettling answer: 7 out of 11 skill evaluations changed scores between two runs, with the largest swing hitting a full point. This wasn't some extreme lab case — it was an accidental natural experiment. The second run was only meant to fix retry logic; nobody was specifically watching for score drift.

Same Pipeline, Two Evaluations, Seven Differences

Every score the team evaluated was generated by a model reading SKILL.md according to a fixed rubric. They put 11 skills through two evaluation rounds: same input files, same model, same rubric, two hours apart. Here's what happened:

Skill	First Run	Second Run	Difference
understand-explain	8.3	9.0	+0.7
presentation-styling	7.3	8.0	+0.7
understand-figma	8.3	8.7	+0.4
understand-onboard	8.0	8.3	+0.3
understand-knowledge	8.3	8.0	−0.3
understand-domain	8.3	8.0	−0.3
presentation-structure	8.3	7.3	−1.0
Remaining 4 skills	—	—	0

The maximum swing was exactly 1 point, and the direction wasn't consistent — some went up, some went down. This means you can't systematically attribute it to "the model got stricter" or "it was having an off day."

An Imperfect but More Revealing Experiment

To be upfront: between the two runs, the team also adjusted the generation budget (the first run occasionally hit the token limit before answering). So some of the variance may come from the execution framework rather than the model's judgment itself. The author chose to disclose this flaw for a straightforward reason: automated scoring that can't be audited is exactly what invites the most dangerous kind of blind trust. This measurement is a natural experiment, not a controlled one, and should be treated as a variance floor.

Natural experiment vs. controlled experiment: A natural experiment is one where the researcher doesn't actively manipulate variables but instead exploits changes that happen to occur in reality to observe differences in outcomes. Its advantage is that it mirrors real-world scenarios at low cost; its disadvantage is that it can't rule out confounding factors. A controlled experiment, by contrast, requires strict variable control — changing only one factor — which enables stronger causal claims. In this article, the team's second run was originally just to fix retry logic, not to deliberately control variables, making it a natural experiment. It can't distinguish whether score fluctuations come from randomness in model judgment, sampling temperature, or changes in generation budget — but precisely because of that, it reflects variance in a real production environment, not a variance floor under idealized conditions. Treating such results as a "variance floor" means the true fluctuation is likely larger, not smaller.

What You Should Take Away from the Data

The data offers lessons more specific than "scores change."

Usable Resolution Is About 0.5 Points

Not 0.1 points. If you're comparing 8.3 and 8.7, you're reading noise regardless of who generated those numbers. What survives a rerun is the band: the difference between the "9 band" and the "7 band" is meaningful; digits after the decimal point are not. Putting a 0.1-point gap in a headline is essentially advertising that you don't understand your own scoring tool.

Measurement resolution and the noise floor: In metrology, resolution refers to the smallest difference a measurement tool can reliably distinguish — signals below that threshold get drowned out by noise. For LLM evaluation, even if the rubric demands one decimal place, each model generation is still influenced by sampling randomness, context order, and internal state, causing different judgments for the same input. When the same skill's score differs by 0.7 or even 1.0 points between runs, it means system noise has exceeded the 0.1-point scale. Therefore, any gap smaller than about 0.5 points should not be interpreted as a real quality difference. This aligns with the concept of "measurement uncertainty": when reporting a score, you must also report its uncertainty range — otherwise readers will mistakenly believe the difference between 8.3 and 8.7 is real. In practice, rounding scores to multiples of 0.5 or using band labels directly is an effective way to reduce misleading interpretations.

Publish the Rationale, Not Just the Number

A bare score gives readers nothing to push back against. Every score from this team comes with written reasoning behind it — if you think presentation-structure shouldn't be just 7.3, you can read the rationale and raise an objection. This is the dividing line between "measurement" and "a feeling with a number attached."

Auditability and scoring rationale: If an automated scoring system only outputs a number, it severs the path from conclusion back to evidence, making errors impossible to locate and disputes impossible to arbitrate. Publishing scoring rationale means disclosing the model's reasoning chain or scoring basis, letting readers judge whether a score is reasonable. This resembles the "reproducibility" requirement in academic peer review: not just results, but the process that supports them. When scores come with rationale, even if they fluctuate due to randomness, readers can understand which dimension of judgment the fluctuation comes from — rather than blindly accepting or wholesale rejecting. For leaderboards, quality gates, and similar scenarios, retaining the reasoning process also enables rapid identification of why scores changed after a model update, avoiding mistaking system noise for quality improvement or degradation.

Design Your Publishing Format Around Variance

Since a rerun can move a score by 1 point, your UI, API, and headlines must all be able to absorb that variance. Use bands; never put a 0.1-point gap in a headline; state "scores are accurate to ±0.5" in your documentation. That's a hundred times more honest than an unqualified number — and it costs you nothing.

Three Self-Check Items for Your Evaluation Pipeline

If you're running any automated scoring, here are three things you can do this week:

1. Rerun 10–20 already-scored items with identical inputs. Almost nobody does this, and the answer is always surprising.
2. Measure your own resolution floor and document it. Explicitly stating your margin of error is far more credible than a bare score.
3. Keep the reasoning process. If your evaluation only outputs numbers, you can't debug, can't appeal, and can't explain when someone challenges you.

The rubric the author used has been made public, including weights and the meaning of each dimension. Whether you're building a leaderboard, a quality gate, or an evaluation system, this dataset points to the same conclusion: the credibility of automated scores depends on whether you honestly present their uncertainty.