Bayesian Optimization on Function Spaces: A Deep Dive into L0MO

From Vectors to Functions: The New Frontier of Bayesian Optimization

Bayesian Optimization (BO) is already a mature technique for tackling black-box function minimization in areas like machine learning hyperparameter tuning and experimental design. Its typical assumption is that the optimization target is a finite-dimensional vector. In practice, however, many parameter vectors are actually discretizations of something fundamentally continuous — think control curves, material distributions, or time-varying signals. Forcing these objects into a fixed-dimensional vector space often discards their inherent structure.

This is precisely the core problem that Functional Bayesian Optimization (FBO) — a growing area of research — aims to solve: the variable being optimized is no longer a member of a finite-dimensional vector space, but lives in an infinite-dimensional function space. A new arXiv paper tackles this setting head-on, proposing a method called L0 Manifold Optimization (L0MO).

Classic Bayesian optimization revolves around two core components: a Gaussian Process (GP) as a probabilistic surrogate model for the objective, and an acquisition function (e.g., Expected Improvement EI, Upper Confidence Bound UCB) that determines the next query point. The computational bottleneck lies in maximizing the acquisition function over the parameter space at each iteration. When the parameter space is extended from finite-dimensional vectors to infinite-dimensional function spaces, this inner optimization problem becomes extremely challenging — exhaustive search is impossible, and gradient descent cannot be directly applied. How to parameterize the function space and reduce it to a tractable finite form thus becomes the central challenge in FBO research.

L0MO: Searching on a Sparse RKHS Manifold

The core idea of the paper is to restrict the search to a particular subset of a Reproducing Kernel Hilbert Space (RKHS): the subset of functions that admit a sparse representation in terms of kernel functions — that is, the target function can be approximated by a linear combination of relatively few kernel functions.

Concretely, L0MO jointly optimizes two sets of quantities:

• Kernel locations: determining where in the domain these basis functions are placed;
• Kernel coefficients: determining the weight of each basis function.

This "sparse representation" design is precisely what the L0 in the method's name refers to — it implies control over the number of nonzero terms in the representation. By restricting the search to a sparse manifold, L0MO retains the expressive power of function-space optimization while avoiding the computational difficulties of operating directly in infinite-dimensional space. Compared to crudely discretizing functions into high-dimensional vectors and applying standard BO, this approach better respects the problem's intrinsic geometric structure.

Reproducing Kernel Hilbert Space (RKHS) is a special class of function spaces with a key property: every function in the space is uniquely determined by a kernel function, and pointwise evaluation is a continuous linear functional. Common kernel functions include the Gaussian (RBF) kernel and the Matérn kernel. RKHS is ubiquitous in machine learning because it combines the expressive flexibility of infinite-dimensional function spaces with an inner product structure, allowing much of the intuition from finite-dimensional linear algebra to carry over. In Bayesian optimization, the covariance function of a Gaussian process itself defines an RKHS — meaning L0MO's search space is mathematically compatible with the priors commonly used in BO. The notion of "sparse representation" is analogous to sparse decomposition in signal processing: a target function is described to sufficient accuracy by a linear combination of a small number of basis functions, compressing the infinite-dimensional problem into a joint optimization over a finite set of parameters (locations + coefficients).

A Unifying Lens: Relationship to Existing Methods

One of the paper's important contributions is not just the new algorithm itself, but the provision of a unifying analytical lens. The authors carefully discuss the relationship between L0MO and existing FBO methods, bringing previously scattered research under a common framework.

This kind of unifying perspective is highly valuable in methodology research: it helps readers see the shared mathematical structure underlying different approaches, and clarifies where the new method fits — L0MO is not a replacement that appeared from nowhere, but a natural extension and generalization of existing ideas. For researchers looking to enter the FBO field, this synthesis serves as a useful reference map in its own right.

New Benchmarks and Experimental Validation

Evaluating a function-space optimization method faces a practical challenge: the lack of suitable test benchmarks. Traditional BO benchmark functions are almost all designed for finite-dimensional vectors.

To address this, the authors construct a new suite of benchmark functions by "porting" standard finite-dimensional test functions to infinite-dimensional domains. This byproduct is itself valuable to the broader FBO community — it provides a reusable testbed for comparing future methods.

Using this extended benchmark suite, the authors conduct a large-scale computational study comparing L0MO against current state-of-the-art methods. Experimental results show that overall, L0MO achieves superior performance across a broad range of benchmarks.

Significance and Outlook

The value of L0MO lies in its use of a relatively clean idea — joint optimization on a sparse RKHS manifold — to address the fundamentally difficult problem of optimization in function spaces. For application domains where parameters naturally arise from continuous functions (such as control, physical modeling, and signal design), optimizing directly at the function level promises more efficient solutions that better respect the problem's structure.

Of course, as a freshly published preprint, questions about the method's robustness on large-scale real-world problems, sensitivity to sparsity level selection, and the computational cost of kernel location optimization remain to be further examined by the community. Nevertheless, the unifying framework and new benchmarks it provides have already injected fresh momentum into the field of functional Bayesian optimization.