The CI Problem with AI Agents: Should Score Fluctuations Block Your PR?

In traditional software testing, a pass is a pass and a failure is a failure — the verdict is always black or white. But when the system under test is an AI Agent, that deterministic logic breaks down completely. The same agent running against the same code can produce entirely different results on two consecutive runs. A developer working on Agent CI shared the pitfalls they encountered on Reddit, along with an open-source solution called AgentSeism.

The Root Cause: Non-Determinism in Agent Evaluation

The developer initially underestimated one critical fact: the same Agent can produce significantly different evaluation results even when the code hasn't changed at all.

They shared a real example: a coding Agent's success rate dropped from 92% to 88% with zero code modifications. Under traditional CI logic, that 4-percentage-point decline would be enough to block the PR from merging. But this was actually a false positive — the score fluctuation was simply a reflection of the Agent's inherent randomness, not a genuine capability regression.

If you simply set a threshold like "block if success rate drops more than X%", normal fluctuations will constantly trigger false alarms, making the entire CI pipeline unreliable.

The Flip Side: Real Regressions Slip Through Undetected

The more troubling scenario is the opposite. The author deliberately degraded the Agent's capabilities, causing the success rate to plummet from 92% to 52% — a genuinely severe regression. Yet the statistical rules they had originally designed let the PR through.

The reason: the regression's damage was concentrated on a small number of tasks rather than distributed evenly across the entire test suite. From the perspective of the overall statistical distribution, the average reliability change across the suite never crossed the threshold, so the rules concluded "no problem."

This contradiction exposed a critical distinction that had been conflated. The author realized they needed to split evaluation into two separate questions:

• Has the overall suite become broadly less reliable? (reliability broadly worse)
• Has a previously stable capability suddenly collapsed? (a capability collapsed)

A single average-based rule can never answer both questions simultaneously. The first requires looking at overall trends; the second requires watching for point-level collapses.

Treating a PR as an Experiment, Not a Deterministic Test

The author's eventual shift in thinking is instructive: stop treating a PR as a deterministic test, and start treating it as an experiment.

This methodology incorporates several core elements from statistical experimental design:

Repeated Runs

Single-run results are unreliable. Multiple runs are necessary to estimate the true distribution of success rates and separate random noise from real signal.

Practical Effect Thresholds

Rather than detecting every tiny difference, set a threshold for effect sizes that are "practically meaningful" — avoiding overreaction to inconsequential fluctuations.

The distinction between Statistical Significance and Practical Significance is one of the core ideas behind this methodology. Traditional hypothesis testing only answers "could the observed difference be caused by randomness?" — not "is this difference large enough to actually matter?" With sufficient sample size, even a 0.1% drop in success rate can achieve statistical significance, yet it's meaningless in practice. Setting a practical effect threshold (also known as a Minimum Detectable Effect, or MDE) requires engineers to define upfront what counts as a real problem — for example, "only trigger a block if success rate drops more than 5 percentage points." This forces teams to think about quality standards in business terms rather than blindly chasing statistical p-values, fundamentally preventing overreaction to an Agent's day-to-day random variation.

Uncertainty & Multiplicity

Explicitly model result uncertainty, and handle the multiple testing problem (multiplicity) that arises when comparing across multiple capability dimensions simultaneously — this is precisely what's needed to catch "single capability collapse" scenarios.

The Multiple Testing Problem is a classic statistical trap: when you test multiple hypotheses simultaneously, even if each individual test has only a 5% false positive rate, the overall false positive rate skyrockets as the number of tests grows. For example, if you simultaneously evaluate 20 capability dimensions, even with no real change in the Agent's abilities, on average one dimension will "appear" to have significantly degraded. Common correction methods include the Bonferroni correction (dividing the significance threshold by the number of tests) and the Benjamini-Hochberg method (controlling the False Discovery Rate, or FDR). This problem is especially acute in Agent evaluation — an Agent typically needs to be assessed across multiple capability dimensions such as code generation, tool use, and reasoning, and testing each dimension independently without correction inevitably generates a flood of false alarms. AgentSeism explicitly addresses this issue to enable reliable detection of genuine point-level collapses even in multi-dimensional evaluations.

An Explicit "Insufficient Evidence" State

This is the most noteworthy design decision. Rather than producing only "pass" or "fail", the system introduces a clear third state: insufficient evidence. When the data cannot support a reliable judgment, the system honestly says "I don't know" rather than forcing a potentially wrong conclusion.

AgentSeism: An Open-Source Agent Regression Detection Tool

The author documented their reasoning and hard-won lessons in a blog post, and open-sourced the tool on GitHub under the name AgentSeism. The "Seism" (earthquake) in the name is apt — the tool's job is to detect real, structural "earthquakes" in an Agent's capability landscape while ignoring the everyday background noise.

For any team running Agents in production and trying to build a CI/CD pipeline around them, this problem is unavoidable. The question the author posed at the end of their post cuts right to the heart of an industry-wide challenge:

If you're running Agents in CI today, how do you decide when a score drop is "real enough" to block a merge?

There's no standard answer yet. The deterministic thinking of traditional software engineering breaks down in the face of probabilistic AI systems. Finding the right balance between "missing a real regression" and "false-alarming on normal variation" is a fundamental infrastructure problem that LLM application engineering must solve.

Implications for AI Engineering

The value of this case study extends beyond the tool itself. It reveals a broader trend: as AI Agents become central to software development workflows, our entire testing and quality assurance paradigm needs to be rebuilt.

The shift from deterministic testing to statistical experimentation, and from binary judgment to three-state output (pass / fail / insufficient evidence), is fundamentally an acknowledgment of the probabilistic nature of AI systems. Any team that tries to force-fit traditional CI thinking onto Agent evaluation will eventually encounter both types of errors the author described. AgentSeism offers not just a tool, but a mental framework worth adopting.