[KongchangAI]
· 2 min read· 1,282 words

LLM-Assisted Multi-Fidelity Bayesian Optimization: Integrating Expert Knowledge with Numerical Data

LLM-Assisted Multi-Fidelity Bayesian Optimization: Integrating Expert Knowledge with Numerical Data

LLMs turn expert qualitative knowledge into a knowledge basis fused with numerical data, accelerating multi-fidelity Bayesian optimization under scarce high-fidelity data.

This article proposes an LLM-assisted multi-fidelity Bayesian optimization (MFBO) method to address the difficulty of accurately modeling the residual between low-fidelity and high-fidelity models when high-fidelity data is scarce. The method uses an LLM to interpret expert textual descriptions of physical mechanisms neglected by the low-fidelity model, quantifies this qualitative knowledge into continuous basis functions (a knowledge basis) via pairwise comparisons and a pairwise Gaussian process, and calibrates and fuses it with numerical data. In a lithium-ion battery design experiment, it reduces the area under the regret curve by 61% versus traditional MFBO; controlled experiments show knowledge quality, not LLM choice, is the key accelerator. The framework offers a new pathway for fusing expert qualitative and quantitative information in engineering optimization.

Introduction

In engineering design, multi-fidelity Bayesian optimization (MFBO) can significantly reduce optimization costs by combining cheap low-fidelity (LF) models with a small number of expensive high-fidelity (HF) evaluations. However, when high-fidelity data is scarce, learning the discrepancy between low-fidelity and high-fidelity models from numerical data alone is very difficult—especially when the physical mechanisms ignored by the low-fidelity model are design-dependent. Despite limited data, domain experts often know which physical mechanisms the low-fidelity model neglects, and under what conditions these mechanisms have a greater impact. Existing methods cannot directly exploit such qualitative knowledge, but large language models (LLMs) can interpret expert descriptions and compare designs accordingly. This article introduces an LLM-assisted MFBO method that converts expert knowledge into a knowledge basis and fuses it with numerical data to accelerate the optimization process.

Schematic of LLM-assisted multi-fidelity Bayesian optimization

Core Challenge: Multi-Fidelity Optimization Under Scarce High-Fidelity Data

Multi-fidelity Bayesian optimization aims to balance the low cost of low-fidelity models with the accuracy of high-fidelity models, finding the optimal design with as few high-fidelity evaluations as possible. In real engineering practice, however, high-fidelity data is often hard to obtain, making the discrepancy between low-fidelity and high-fidelity (i.e., the residual) difficult to model accurately. More challenging still, when the physical phenomena ignored by the low-fidelity model (such as certain nonlinear effects) are correlated with design parameters, the residual is no longer a simple constant offset but a complex function that varies with the design. In this case, a multi-fidelity model trained only on limited numerical data is prone to bias, which degrades optimization efficiency.

Bayesian optimization is a global optimization method based on probabilistic surrogate models (typically Gaussian processes), particularly suited to black-box optimization problems where objective function evaluations are expensive and gradient information is unavailable. It builds a surrogate model to approximate the objective function and uses an acquisition function (such as expected improvement, EI, or upper confidence bound, UCB) to trade off exploration and exploitation, thereby finding the optimal solution with as few evaluations as possible. In multi-fidelity Bayesian optimization, the surrogate model fuses data from different fidelity levels simultaneously: low-fidelity data is abundant but less accurate, while high-fidelity data is scarce but more accurate. Typical multi-fidelity models (such as autoregressive Gaussian processes) assume that the high-fidelity function can be expressed as the low-fidelity function multiplied by a scaling factor plus a residual function, with the residual also typically modeled as a Gaussian process. When high-fidelity data is scarce, the posterior uncertainty of the residual function is large, making acquisition-function decisions unreliable—precisely the key pain point this article addresses.

The LLM-Assisted MFBO Framework

Constructing the Knowledge Basis

The core of the method is to add a "knowledge basis" on top of the traditional multi-fidelity model. Specifically, starting from the expert's textual description of the neglected physical mechanisms, an LLM is used to judge, for two designs, which one requires a larger signed residual correction. These pairwise judgments are converted into continuous basis functions via a pairwise Gaussian process, thereby quantifying qualitative knowledge into prior information usable by the model.

Pairwise Gaussian process (pairwise GP) is a probabilistic model based on pairwise comparison data. Unlike a traditional Gaussian process that directly models function values, the observations of a pairwise GP are relative comparison information such as "is the response of design A greater than that of design B." Its core idea is to assume a latent function f, with the observed pairwise comparison outcome determined by the sign of f(A)-f(B), and a link function (such as the probit function) mapping the difference to the probability of the comparison outcome. Given several pairwise comparisons, the posterior distribution of the latent function f can be obtained via Laplace approximation or variational inference. This approach is widely used in psychometrics (e.g., preference learning) and ranking problems. In this article, the LLM's judgment of which of two designs requires a larger residual correction is essentially a pairwise comparison, so the pairwise GP naturally serves as the mathematical bridge that converts qualitative judgments into continuous basis functions.

Calibration and Fusion

To avoid blindly trusting expert knowledge, high-fidelity data is used to calibrate the magnitude and sign of the knowledge basis's contribution. In this way, the knowledge basis and numerical data work in concert, leveraging the expert's qualitative insights while ensuring the model's adaptability to data.

Schematic of LLM-assisted multi-fidelity Bayesian optimization

Experimental Validation: Lithium-Ion Battery Design Problem

On a lithium-ion battery design problem, this method reduces the area under the regret curve by 61% compared with traditional MFBO. This significant improvement shows that expert knowledge interpreted by the LLM can effectively supplement numerical data and accelerate the optimization search. In addition, a controlled experiment that independently varied knowledge quality and LLM choice showed that the key factor determining search acceleration is knowledge quality rather than the specific LLM choice. A knowledge-level ablation study further quantified the contribution of each knowledge statement.

Area Under the Regret Curve (AURC) is a commonly used metric for measuring the efficiency of optimization algorithms. Regret is defined as the gap between the objective value of the current best solution and the global optimum; as iterations proceed, an ideal optimization algorithm should make the regret decrease rapidly. Integrating the curve of regret versus number of evaluations yields the AURC; a smaller value indicates that the algorithm approaches the optimal solution faster. Compared with looking only at the final best value, the AURC comprehensively reflects the convergence speed of the optimization process and avoids the contingency of "getting lucky and finding the optimum in one shot." The reported 61% reduction in AURC means that, under the same evaluation budget, the near-optimal solutions found by the LLM-assisted method are significantly better than those of traditional MFBO, and the advantage persists throughout the optimization process rather than appearing only at the endpoint.

Multi-Source Perspectives and Implications

Although the current analysis is mainly based on a single source, this research reveals a new pathway for fusing qualitative and quantitative information. The combination of expert knowledge and numerical data is not a simple superposition but rather a complementarity achieved through the LLM's semantic understanding capabilities. In the future, this approach is expected to generalize to other engineering fields that rely on expensive simulations, such as aerospace and materials design.

Conclusion

LLM-assisted MFBO effectively alleviates the optimization challenge under scarce high-fidelity data by formalizing expert knowledge into a knowledge basis and calibrating and fusing it with numerical data. Experiments demonstrate that knowledge quality is the key to accelerating the search, while the choice of LLM has little effect. This framework offers a new perspective for human-machine collaboration in engineering optimization.

One-line Summary

LLM converts expert qualitative knowledge into a knowledge basis and fuses it with numerical data to accelerate multi-fidelity Bayesian optimization under scarce high-fidelity data.

Paragraph Summary

This article proposes an LLM-assisted multi-fidelity Bayesian optimization (MFBO) method aimed at solving the difficulty of accurately modeling the residual between low-fidelity and high-fidelity models when high-fidelity data is scarce. The method uses a large language model to interpret expert textual descriptions of physical mechanisms neglected by the low-fidelity model, determines via pairwise comparison the direction of residual correction required by different designs, and then quantifies the qualitative knowledge into continuous basis functions (a knowledge basis) via a pairwise Gaussian process, calibrating and fusing it with numerical data. In a lithium-ion battery design experiment, this method reduces the area under the regret curve by 61% compared with traditional MFBO. Controlled experiments show that knowledge quality, rather than the specific LLM choice, is the key factor accelerating the search. The framework provides a new pathway for integrating expert qualitative and quantitative information in engineering optimization.

#multi-fidelity Bayesian optimization#LLM-assisted optimization#expert knowledge fusion#high-fidelity data scarcity#lithium-ion battery design#pairwise Gaussian process
Share:

Related articles