Kriging Reliability Analysis: Inner Optimizer Comparison and the Risk of Missing Failure Modes

Active learning Kriging in reliability analysis often replaces candidate-pool selection with continuous metaheuristic search. A pre-registered comparative study, holding the surrogate model, population, and stopping rule constant, isolated the effect of the inner optimizer and revealed non-conservative estimation problems caused by missing failure modes.

Study Design and Core Hypotheses

The study compared five budget-matched continuous optimizers (including a hybrid population optimization with L-BFGS-B refinement) against a candidate-pool method across 11 problems (dimensions from 2 to 100). The primary hypothesis was not supported: in comparisons with optimizers using the same criterion, the hybrid method won 6, tied 41, and lost 19 out of 66 Holm-corrected accuracy comparisons; against the pool method, it won 0, tied 23, and lost 21 out of 44.

This result directly challenges the intuition that "continuous optimizers are necessarily better than candidate pools." Although the hybrid method performed well in some scenarios, it showed no significant overall advantage, indicating that the choice of inner optimizer requires more careful evaluation.

L-BFGS-B is a quasi-Newton method, a limited-memory variant of the Broyden–Fletcher–Goldfarb–Shanno algorithm that supports bound constraints on variables. It approximates the Hessian matrix using gradient and displacement information from the most recent steps, without storing the full Hessian, making it suitable for medium- to high-dimensional continuous optimization problems. In active learning Kriging, it is often used as the inner optimizer to maximize the learning function (e.g., an error criterion). In this study, "hybrid population optimization" refers to combining a swarm-intelligence algorithm with L-BFGS-B refinement: first using population optimization for global exploration, then L-BFGS-B for fine local search, balancing globality and convergence speed. This design aims to compensate for the tendency of pure gradient methods to get stuck in local optima, but its effectiveness still depends on the properties of the learning function's landscape.

Holm correction is a multiple-comparison correction method used to control the family-wise error rate (FWER). When multiple hypothesis tests are performed simultaneously, the probability of false positives accumulates without correction. The Holm method orders p-values in ascending order and compares them sequentially against decreasing significance thresholds, stopping once a test is not significant. The "66 Holm-corrected accuracy comparisons" in the text means that strict multiple-testing control was applied to multiple paired comparisons to reduce the risk of chance significance. The win/tie/loss counts are tallied from these corrected comparisons, making the conclusions more conservative and reliable. This pre-registration plus multiple-correction design enhances the statistical rigor of the comparative study.

Failure Mode Missing and Underestimation Risk

In sequential selection for three series systems, multi-start L-BFGS-B failed to learn a certain failure mode in 59 of 60 runs, a higher frequency than population optimization, genetic algorithms, or AK-MCS. Across all 198 runs (including 22 post-freeze), when an unlearned failure mode existed, the learned modes failed to compensate, and the failure probability was underestimated (exploratory analysis).

The error criterion may accept significant underestimation: it accepted failure probabilities as low as 0.38 to 0.89 times the reference value in 17 runs (13 of which were pool selection); another 181 runs exhausted the budget. Follow-up analysis on single-design-point problems showed that underestimation is likely when selection concentrates near the design point.

Failure modes refer to the multiple paths by which a system fails under different input regions or different physical mechanisms. In a series system, failure of any component causes system failure, so multiple failure domains (modes) exist. If active learning Kriging learns only some failure modes, the surrogate model will incorrectly predict high reliability in unlearned regions, thereby underestimating the failure probability. Sequential selection means selecting one design point per round to update the surrogate model, rather than generating a candidate pool all at once. Multi-start L-BFGS-B searches for the maximum of the learning function from multiple initial points, but if the learning-function peaks corresponding to some failure modes are narrow or masked by other modes, the optimizer may repeatedly converge near already-learned modes, causing missed learning. AK-MCS is the classic method combining active learning Kriging with Monte Carlo simulation, using candidate-pool selection. Coverage checking means verifying after stopping whether the surrogate model has sufficient accuracy across all candidate regions (especially near unlearned modes), for example by checking prediction variance or comparing against reference simulations, to detect potential missed learning and trigger supplementary sampling.

Advantages of Candidate-Pool Selection and Recommendations

Kriging-believer pool selection satisfied the surrogate stopping criterion with fewer calls than the best continuous method in all four information-rich batch units (two of which were post-freeze). Based on this, the authors recommend adopting pool selection by default and performing coverage checks for any selection method.

Practical Implications

This study reminds engineers and researchers that the choice of inner optimizer is not irrelevant: continuous optimizers may lead to non-conservative reliability estimates by missing failure modes. In the active learning Kriging framework, prioritizing candidate-pool methods and adding coverage checks is an effective strategy to reduce risk.

One-line Summary

Pre-registered comparison finds continuous optimizers no better than candidate pools and prone to missing failure modes, causing underestimation.

Paragraph Summary

This pre-registered study compared five budget-matched continuous optimizers with a candidate-pool method in active learning Kriging reliability analysis, holding the surrogate model, population, and stopping rule constant. The primary hypothesis was not supported: the hybrid continuous optimizer did not significantly outperform same-criterion optimizers or the pool method in Holm-corrected accuracy comparisons. More critically, multi-start L-BFGS-B frequently missed failure modes in sequential selection for series systems, leading to underestimated failure probabilities; the error criterion even accepted estimates as low as 0.38–0.89 times the reference value. In contrast, Kriging-believer pool selection met the stopping criterion with fewer calls in information-rich batches. The authors recommend adopting pool selection by default and adding coverage checks to reduce the risk of non-conservative estimates.