The False-Positive Crisis of Drift Detectors: Three Lessons from HivePlane v0.2.0

In building a reliability assurance system for AI Agents, drift detection is often regarded as a core line of defense. But the author of the HivePlane project revealed a counterintuitive dilemma in version v0.2.0: a detector that cries wolf is worse than no detector at all. Because every false positive trains operators to ignore the signal, and a gate that is ignored is no longer a gate.

When a larger benchmark set is misjudged as capability decay

The HivePlane v0.2.0 drift detector was originally designed to compare an agent's failure count across different corpus versions. The problem: when the learning loop incorporates a newly reviewed use case and the benchmark set grows from 10 tasks to 20, an agent that previously passed 10/10 suddenly shows "10 failures" under the new counting method. The author wrote in the release report: "The agent didn't get worse — the benchmark set got bigger."

The detector could not distinguish "corpus growth" from "capability decline," so a perfectly healthy agent was automatically quarantined. This finding, numbered #530, changed the entire team's understanding of drift baselines.

The fix: drift must be measured against the agent's own baseline on a comparable corpus. What is compared is the same batch of tasks across different re-evaluations, not failure counts on corpora of different sizes. A corpus version change establishes a new baseline, not a drift signal. A larger benchmark set is growth, not regression.

Drift Detection in the context of machine learning and AI operations refers to the process of continuously monitoring whether the behavior distribution of a model or agent deviates from an established baseline. The traditional approach compares evaluation metrics on a fixed dataset over time, but this paradigm carries an implicit assumption: that the evaluation set itself is stable. Once the evaluation set grows dynamically (for example, continuously incorporating newly reviewed use cases), absolute metrics such as failure counts become distorted. A more robust approach is to use a comparable baseline — repeatedly evaluating on the same batch of tasks and the same corpus version, comparing only differences within the same set before and after. Baseline updates (such as a corpus version upgrade) should be treated as resetting the frame of reference, not as drift events. This also explains why many production-grade monitoring systems distinguish "data drift," "concept drift," and "evaluation set changes" — only the first two should trigger alerts.

The recovery mechanism rejected its own quarantine order

The second problem (numbered F-13) is even more ironic. Automatic drift quarantine resets the certification status to UNCERTIFIED rather than QUARANTINED. But the recovery logic requires status == QUARANTINED to execute. The result: the gate that creates quarantines refuses to lift them. A workload can be automatically quarantined but can never recover, because the precondition describes a state the detector never sets.

The fix was to remove the outdated precondition. The recovery logic now checks for active quarantine records rather than the certification status enum — quarantine records are the source of truth for "whether it is quarantined." This defect was caught in live testing in scenario S5, right after the automatic quarantine in S4.

Status Enums and Event Records are two common source-of-truth models in state management. A status enum compresses the current state into a single field value (such as UNCERTIFIED or QUARANTINED) — concise but prone to drifting out of sync with history; an event record preserves the complete trajectory of every state change and derives the current state by aggregation at query time. In this case, the drift detector wrote to the certification status field, while the recovery logic read the quarantine status enum — the semantic mismatch left the "quarantine order unclaimed." Using quarantine records as the source of truth means that any path that writes a quarantine leaves a record that the recovery logic can recognize, thereby avoiding unreachable branches in the state machine. This pattern is known as "log-centric state management" in distributed systems and workflow engines.

The probe cannot smell the decay signal

The third problem (#609 / S24) is the most subtle. The synthetic probe runner ignored the Spec.expected field and only checked whether the run was admitted. This means the probe could only fail when a run was rejected; it could never flag "the agent ran but gave a wrong answer." The author offered a sharp formulation: a probe is a liveness probe, not a readiness probe.

The probe's job is to catch decay before the drift threshold triggers. If the probe only checks "whether it runs," it cannot detect "whether the run is correct." After the fix, the probe starts a run, waits for termination, and compares the output against expected. A mismatch triggers a probe.decay warning, and at that point the probe can smell decay earlier than the drift detector.

Readiness Probes and Liveness Probes originate from Kubernetes health-check semantics: a readiness probe determines whether a container can receive traffic, and a liveness probe determines whether a container is still running. Both answer only "can it be used," not "is it being used correctly." In AI Agent scenarios, this is far from enough — an agent may run normally yet output wrong answers. Hence the need for a third type of probe: an Assertion Probe, which performs a complete inference and compares against the Spec.expected field. Only when a probe has output-assertion capability can it catch semantic-level decay signals before the drift threshold triggers. This is also the technical foundation of the claim that "a liveness probe is not a readiness probe": the layers of health checks must match the layers of the system's failure modes.

False-positive control is the product itself

The author concluded with four principles: a single exceedance is a warning, not a quarantine — quarantine requires N consecutive failures or a strong signal, so a stable agent is never quarantined due to noise; drift must be measured against a comparable baseline — the same tasks, the same corpus version; quarantine records rather than status enums are the source of truth; and probes must assert expected output — a liveness probe is not a readiness probe.

The v0.2.0 live tests demonstrated the complete loop through scenarios S4, S5, S8, and S24 — drift detection, evaluation, probes, automatic quarantine, and recovery after re-certification. The 36/36 test-pass record and complete design documentation are publicly available in the project repository.

If your operations team has already learned to execute recovery operations without reading them, then your detector is not a control — it's just a checkbox.

One-line Summary

False positives from drift detectors train operators to ignore signals; HivePlane v0.2.0 fixes three anti-patterns with comparable baselines, quarantine records, and assertion probes.

Paragraph Summary

HivePlane v0.2.0 reveals a counterintuitive dilemma in AI Agent reliability assurance: a drift detector that cries wolf is worse than no detector at all, because false positives train operators to ignore signals. The article dissects three specific defects: failure-count-based drift detection misjudges benchmark growth as capability decline; the recovery logic requires QUARANTINED status while the quarantiner writes UNCERTIFIED, making quarantines impossible to lift; and synthetic probes only check run admission without verifying expected output, failing to smell decay early. The fixes are to measure drift against the agent's own baseline on a comparable corpus, to use quarantine records rather than status enums as the source of truth, and to have probes assert Spec.expected output. From this the author distills the core principle that false-positive control is the product itself.