Product-of-Experts Gaussian Processes: A New Paradigm for Large-Scale Bayesian Optimization

Large-scale Bayesian optimization (BO) has long faced an engineering dilemma: the approach of independent local Gaussian processes (GP-ind) paired with trust regions can scale to high-dimensional spaces, but it produces discontinuities at region boundaries, loses long-range correlations, and causes a sharp spike in computational overhead. A new paper proposes the BO-pro-c-tr algorithm, which attempts to reconstruct this paradigm at the model level using an "uncertainty-calibrated product-of-experts Gaussian process" (GP-pro-c).

Why Combining Local GP Models Hits a Bottleneck

The core idea of GP-ind is to partition the global problem into several independent subregions, train a local GP in each, and then use a trust region mechanism to decide where to sample. This design seems intuitive, but the paper points out several structural flaws.

The most direct problem is boundary discontinuity: models in disjoint regions know nothing about each other, so predictions jump at boundaries, introducing artificial artifacts into the acquisition function's choices. Second is the absence of long-range spatial correlations—each local model only cares about its own patch, and information at the global level cannot propagate upward, potentially causing information loss. In addition, the aggressive expansion of local models triggers a steep rise in computational overhead, undermining the efficiency advantage that a divide-and-conquer strategy should bring.

These flaws cannot be fixed by tuning parameters; they stem from the modeling assumption of "independent local models" itself.

Gaussian Process (GP) is the standard surrogate model in Bayesian optimization. It defines a prior distribution over functions, where the function values at any finite set of points follow a joint Gaussian distribution, fully characterized by a mean function and a covariance function (kernel function). The advantage of a GP is that it provides both a predictive mean and variance, with the latter serving directly as the source of uncertainty estimates in acquisition functions. However, training a GP requires computing the inverse of the covariance matrix, with a time complexity of O(n³), where n is the number of observation points. When observations reach thousands or even tens of thousands, this overhead becomes unacceptable. Independent local GPs were created precisely to circumvent this bottleneck: splitting the search space into multiple subregions, each training a local GP on only a small amount of data from its own region, thereby keeping the cubic complexity at a manageable local scale. But this creates a fragmentation at the model level, because each local model sees only its own data, and the smoothness and correlation assumptions of the global function are broken.

Product-of-Experts Models: A Modeling Shift from Independent to Joint

The idea of POE is to have multiple experts (i.e., multiple local GPs) jointly produce distributions for the same prediction point, and then fuse them via multiplication into a single Gaussian prediction. Compared with the hard switching of "first select a region, then look at the local model," POE naturally possesses smooth fusion properties, which can alleviate boundary discontinuity and allow information from different regions to converge at the prediction level.

The paper adds uncertainty calibration on top of standard POE, forming GP-pro-c. This calibration step targets a classic weakness of POE: after multiple experts predict independently and multiply directly, it is easy to produce overconfident variance estimates, and acquisition functions (such as GP-LCB) rely precisely on uncertainty to decide the direction of exploration. Calibrated uncertainty is more trustworthy, and acquisition decisions are more robust.

It is worth noting that POE is not an entirely new concept, but combining it with trust region mechanisms in large-scale BO scenarios requires solving the engineering problem of "how local models correspond to regions," which is another contribution of the paper.

Product of Experts (PoE) was originally proposed by Hinton. Its core idea is to have multiple models (experts) each produce a probability distribution for the same input, then multiply these distributions and normalize them to obtain a fused predictive distribution. For Gaussian distributions, the product is still Gaussian: its precision matrix (the inverse of the covariance) equals the sum of the experts' precision matrices, and the mean is a precision-weighted average of the experts' means. This means the variance after PoE fusion is usually smaller than that of any single expert—i.e., the consensus of multiple models brings higher confidence. But this is also the source of overconfidence: if the experts are trained on highly overlapping or correlated data, their independence assumption does not hold, and simple multiplication will underestimate the true uncertainty. Uncertainty calibration therefore becomes a necessary step; common approaches include temperature scaling, variance inflation, or posterior calibration based on validation data, with the goal of making the fused variance better match actual prediction errors, thereby providing a reliable exploration signal for the acquisition function.

One-to-Many Association Strategy: Connecting POE with Trust Regions

Traditional GP-ind and trust regions have a "one-to-one" relationship—one local model serves one region. But a surrogate model in POE form is globally shared and cannot simply be mapped this way.

The paper proposes a one-to-many association strategy to solve this alignment problem: a single GP-pro-c surrogate model can support multiple trust regions, with the trust regions still responsible for scheduling local exploration, but the underlying predictions provided by a unified POE model. This preserves the advantages of trust regions for local search while avoiding the information fragmentation between independent models.

Architecturally, this is a design that "separates the scheduling layer from the modeling layer": trust regions manage the rhythm of exploration, while POE manages global information fusion.

Trust Region methods are a commonly used local search mechanism in large-scale Bayesian optimization: define a spherical or hyperrectangular region around the current best candidate point, optimize the acquisition function and sample within that region, then expand or shrink the region based on the observed results. In traditional approaches, each trust region corresponds to an independently trained local GP, and when the region switches, the model switches with it, causing prediction discontinuity. The key to the one-to-many association strategy is decoupling: the POE surrogate model continuously updates globally, providing fused predictions for any query point; the trust region is only responsible for deciding "where to sample next," a scheduling problem. Since the query point can fall in any region, POE naturally fuses information from all experts near that point, and predictions no longer jump at boundaries. This architecture is similar to splitting "model learning" and "search control" into two independent modules, making it easier to optimize and scale them separately.

Experimental Data: Modest Accuracy Gains and Significant Overhead Reduction

The empirical results in the paper are noteworthy. When combined with the GP-LCB acquisition function, BO-pro-c-tr achieves a 2.6% average reduction in simple regret and a 34.9% reduction in computational overhead compared with methods based on independent local GPs and trust regions.

The two numbers do not carry equal weight. The 2.6% regret improvement is a solid but modest accuracy gain, indicating that the information sharing brought by POE fusion is indeed effective, but without a disruptive performance leap. The 34.9% overhead reduction is more practically significant: the bottleneck for deploying large-scale BO is often not "whether it can find a better point" but "whether it can afford to run." Saving nearly one-third of the compute means that, under the same budget, larger-scale problems can be covered, or resources can be directed toward more iterations.

It should be noted that these are average results; the paper does not disclose the distribution across different problem sizes or dimensions, so the benefits in actual deployment may vary by scenario.

Simple Regret is a commonly used metric in Bayesian optimization for measuring performance, defined as the gap between the currently recommended best point and the true global optimum, i.e., f(x*) − f(x_best), where x_best is the best solution found by the algorithm so far. Unlike cumulative regret, simple regret focuses only on the quality of the final solution and does not penalize the exploration cost during the search, making it more suitable for evaluating the goal of "how good a solution can ultimately be found." Average simple regret is the average over multiple random seeds or independent runs, used to eliminate the randomness of a single experiment. A 2.6% reduction means that under the same evaluation budget, the final solution found by BO-pro-c-tr is on average closer to the true optimum, but the magnitude is limited, indicating that the information sharing brought by POE fusion mainly plays a supplementary role. In contrast, the reduction in computational overhead directly reflects the number of iterations that can be run or the problem size that can be handled under the same compute, which is often more critical for practical deployment.

Implications for Large-Scale BO Practitioners

The value of this work lies not in inventing some isolated trick, but in pointing to a model-level improvement path: when the boundary effects of a divide-and-conquer strategy become the bottleneck, rather than continuing to patch the trust region scheduling, it is better to reexamine how the surrogate model itself is combined.

For teams that need to perform hyperparameter search, experimental design, or engineering optimization in high-dimensional spaces, the combination of POE and uncertainty calibration offers a compromise—retaining the parallelism of local exploration while restoring a degree of long-range information flow. The cost is that POE's product fusion itself has computational cost; the net overhead reduction reported in the paper shows that this trade-off is worthwhile under its experimental settings, but it still needs to be validated when transferred to other problem structures.

One-line Summary

A new paper proposes BO-pro-c-tr, replacing independent local GPs with a calibrated product-of-experts GP to achieve smoother large-scale Bayesian optimization.

Paragraph Summary

In large-scale Bayesian optimization, the approach of independent local Gaussian processes paired with trust regions suffers from three structural flaws: boundary discontinuity, missing long-range correlations, and a sharp increase in computational overhead. A new paper proposes the BO-pro-c-tr algorithm, which uses an uncertainty-calibrated product-of-experts Gaussian process (GP-pro-c) as the surrogate model, fusing predictions from multiple local experts via multiplication to alleviate boundary jumps and restore information sharing. To connect with the trust region mechanism, the paper designs a one-to-many association strategy, allowing a single POE surrogate model to support multiple trust regions and separating the scheduling layer from the modeling layer. Experiments show that when combined with the GP-LCB acquisition function, simple regret is reduced by an average of 2.6% and computational overhead by 34.9%, with the overhead savings being the more practically significant result.