34 Model Iterations in Review: Most Problems Were Actually in the Evaluation Pipeline

When Model Improvement Becomes Evaluation Debugging

In real-world machine learning engineering, we tend to focus our attention on model architecture, feature engineering, and hyperparameter tuning. However, an engineer from Anexum recently shared a thought-provoking experience on Reddit: he built a "bounded improvement loop" for the AIOps system ANXEngine, and after 34 fully documented iterations, discovered that the vast majority of valuable insights weren't about model improvements at all—they were a series of bugs in the evaluation pipeline.

ANXEngine is an operations system used to perform daily event risk ranking for approximately 200 network devices. The improvement loop the author designed was quite disciplined: register a hypothesis → implement the minimal testable change → train a challenger model and an unchanged control model → evaluate both → make a final decision of KEEP, REVERT, or PAUSE. The process itself demonstrates excellent engineering discipline, but its true value lies precisely in the evaluation pitfalls it exposed.

The methodological foundation of the "bounded improvement loop" comes from hypothesis-driven scientific experimentation. Unlike unconstrained hyperparameter searches or automated NAS (Neural Architecture Search), it emphasizes that each iteration must have a clear hypothesis statement, minimized variable control, and traceable decision records. The challenger-control pattern borrows from the champion/challenger deployment strategy widely used in financial risk management and recommendation systems. Its core principle is ensuring that every change has an unchanged baseline for reference, thereby distinguishing "effects caused by the change" from "random fluctuation."

Three Fatal Evaluation Bugs

Data Leakage: The 0.76 Illusion Drops Back to 0.51

The first problem was overlap between training data and the subsequent "fitness window." In the initial version, the PR-AUC metric looked quite impressive, reaching 0.72 to 0.76. This number was good enough to make one believe the model was performing exceptionally well.

However, when the author strictly separated the training window, gate window, and report window, PR-AUC plummeted to approximately 0.51—nearly random. In the author's own words, this result was "painful, but honest." This is the most classic yet insidious manifestation of data leakage: even a slight misstep in window partitioning for time-series data can dramatically inflate evaluation results.

It's important to understand why PR-AUC (Precision-Recall Area Under Curve) was chosen as the primary evaluation metric. In AIOps scenarios, genuine risk events (positive samples) typically represent only a tiny fraction of total events. Compared to ROC-AUC, PR-AUC is not diluted by the correct classification of large numbers of negative samples, making it a more truthful reflection of the model's ability to detect rare events. Data leakage in time-series scenarios is particularly insidious—simple random splitting breaks temporal causality, and future information can seep into the training set through sliding window features, lag variables, or shared time periods. The standard solution is strict walk-forward validation, ensuring training data is strictly earlier in time than validation data, with a sufficient gap period between them to prevent information leakage from short-term autocorrelation.

Holdout Set Reuse: The "Integrity Crisis" After 13 Experiments

The second problem is more subtle, involving holdout set reuse. The author emphasized that he never trained models on the holdout set, but every evaluation result influenced the direction of the next hypothesis. After running 13 experiments on the same window, claiming that window was "untouched" was no longer honest—because it had indirectly guided the model's evolutionary path.

The essence of this problem is "adaptive overfitting," formally articulated by Dwork et al. in their seminal 2015 paper The Reusable Holdout. When analysts repeatedly adjust models based on validation set feedback, even without ever directly training on the validation set, the model selection process itself gradually fits the idiosyncratic noise of the validation set. This is analogous to the multiple comparisons problem in statistics—each additional comparison inflates the overall false positive rate. Leaderboard overfitting in Kaggle competitions is a classic manifestation: participants' scores on the public leaderboard are often higher than their final private leaderboard scores because their submission strategies have indirectly adapted to the public test set.

To address this, the author introduced an explicit constraint: a maximum of 8 selection queries per window. After that threshold, the window can only be retained as historical evidence and can no longer be used to screen new candidate models. Model promotion requires stricter conditions: either validation passing on two mutually independent positive windows, or one positive selection result plus a completely fresh data slice that has never been queried.

Metric Conflict: Better Ranking, Fewer Early Warnings

The third problem touches on a core trade-off in AIOps scenarios. A certain feature family improved a "difficult device's" risk ranking from position 37 to position 7, while raising Precision@10 from 0.50 to 0.70—by these numbers alone, this was clearly a successful improvement.

But at the same time, the number of truly effective early warnings dropped from 4 to 2. In other words, the model's ranking ability improved, but its ability to detect problems in advance actually degraded. The author ultimately rejected this change with clear reasoning: better ranking isn't worth sacrificing lead time. This decision reveals an important principle—improvement in a single metric does not equal improvement in overall system value.

Precision@K is a commonly used metric in information retrieval and recommendation systems, measuring the proportion of relevant results among the top K ranked results. In this scenario, Precision@10 means that when the operations team focuses on the top 10 ranked devices each day, how many of them actually have problems. However, early warning concerns a different dimension—lead time, i.e., whether the model can signal sufficiently early before an event actually occurs. These two objectives may be fundamentally in conflict: improving ranking precision often means the model relies more heavily on strong signal features that appear close to event occurrence, and these features may not yet be visible in the early stages of an event. This trade-off exists in medical diagnostics, financial fraud detection, and similar domains—a high-precision diagnostic model may only give definitive judgments in late-stage disease, while early screening models need to tolerate higher false positive rates in exchange for a time window.

"Keep" Does Not Equal "Improved"

Another key reflection the author raised: he no longer treats a KEEP decision as proof of model improvement. In a recent audit, the model's comparison results on new data were inconclusive. The model was kept solely because there wasn't a strong enough reason to roll it back, not because performance improvement was confirmed.

This is a concept extremely easy to confuse in MLOps practice. "No reason to roll back" and "confirmed improvement" are two entirely different states, but in many model registries they tend to be recorded as the same "in production" status. A model registry is a core component of MLOps infrastructure, responsible for recording model versions, metadata, lineage, and lifecycle states. Mainstream tools like MLflow Model Registry, Vertex AI Model Registry, and AWS SageMaker Model Registry typically define states like Staging, Production, and Archived, but these states often only reflect deployment decisions without distinguishing the confidence level behind those decisions. Conflating the two leads to so-called "confidence debt"—as multiple inconclusive KEEP decisions accumulate, teams may mistakenly believe the current production model has been thoroughly validated, when in reality it may just be a stack of results that were "not bad enough to replace." Explicitly distinguishing between the two prevents teams from building false confidence in model capabilities.

Unresolved Engineering Challenges

The author candidly admits that the part he's least confident about is query budget design. "8 times" is a practical heuristic limit, not a theoretical value that can be cleanly derived from the data. Since labels arrive slowly, continuously creating new evaluation windows is costly, creating a natural tension between data reuse and freshness.

He posed three open questions to the community:

• Is a fixed query budget reasonable? Or should more rigorous statistical techniques like sequential testing or reusable holdouts be adopted?
• When positive labels arrive slowly, how do you create fresh evaluation data?
• In model registries, should "no reason to roll back" and "confirmed improvement" be distinguished differently?

These three questions hit virtually all the pain points of MLOps systems operating under small-sample, slow-feedback scenarios.

Sequential testing is a branch of statistical hypothesis testing that allows continuous testing as data arrives incrementally, rather than waiting for a fixed sample size to be collected. Common methods include SPRT (Sequential Probability Ratio Test) and more modern always-valid p-values. In MLOps scenarios, sequential testing allows teams to dynamically decide whether to continue an experiment as new data arrives, balancing statistical rigor with resource efficiency. The representative reusable holdout technique—the Thresholdout algorithm proposed by Google Research—works by injecting calibrated noise into each query result, preventing analysts from precisely fitting the holdout set through repeated queries. The algorithm theoretically guarantees that even with an exponential number of adaptive queries, cumulative overfitting risk can be controlled within predetermined bounds, at the cost of slightly reduced precision for each query result. These two techniques may provide theoretically grounded alternatives to fixed budget approaches.

Implications for MLOps Practitioners

The value of this practice log lies not in delivering a better model, but in honestly revealing an often-overlooked truth: in many machine learning projects, the quality of evaluation infrastructure deserves more investment than the model itself. A leaky evaluation pipeline will continuously mislead the entire team's decision-making, and the conclusion that "most gains across 34 iterations came from evaluation bugs" demonstrates just how scarce solid evaluation discipline really is.

For teams building production-grade ML systems, several points can be directly adopted: strict temporal window separation, explicit constraints on holdout query counts, holistic trade-off thinking that rejects single-metric optimization, and distinguishing between "keep" and "improved" semantics in model registries. These seemingly tedious engineering constraints are precisely the foundation that makes AIOps systems trustworthy over the long term.

Key Takeaways