Kernel Methods + Koopman Operator: Bilevel Optimization Achieves New Breakthrough in Data-Driven Modeling

Koopman operator theory provides a linear analysis framework for nonlinear dynamical systems and has become a core tool in data-driven modeling. However, a long-standing challenge for researchers is that when computing finite-dimensional approximations using methods such as Extended Dynamic Mode Decomposition (EDMD), the dictionary must be specified in advance. In recent years, machine learning methods have broken through this limitation by learning dictionaries from data, but mainstream approaches almost all rely on artificial neural network autoencoder architectures.

A new paper proposes EDMD-kDL (Extended Dynamic Mode Decomposition with Kernel-based Dictionary Learning), which for the first time systematically combines kernel methods with bilevel optimization to learn finite-dimensional Koopman embeddings directly from data.

Why Kernel Methods Deserve Attention

Kernel methods have inherent advantages in the interpretability and tractability of theoretical analysis, yet they have been largely overlooked in the context of Koopman embedding learning. The research team's core insight is that kernel dictionary learning can be achieved through collocation points and a bilevel optimization framework, without relying on end-to-end training of neural networks.

This approach combines the ideas of collocation methods and bilevel optimization—the outer optimization is responsible for learning the kernel dictionary, while the inner optimization solves the corresponding Koopman approximation. The two iterate alternately, ultimately yielding both the dictionary and the operator approximation simultaneously.

Kernel Methods and the Kernel Trick: The core idea of kernel methods is to implicitly map data to a high-dimensional (or even infinite-dimensional) feature space through a kernel function $k(x, x')$, without explicitly computing that mapping. According to Mercer's theorem, any kernel function satisfying positive definiteness conditions corresponds to an inner product in some feature space, which allows nonlinear problems to be transformed into linear problems in high-dimensional space. Common kernel functions include the Gaussian radial basis function (RBF) kernel, polynomial kernels, and Laplacian kernels. Compared to neural networks, which require designing the number of layers, activation functions, and training strategies, the choice of kernel methods typically boils down to determining the kernel function type and hyperparameters. Theoretical analysis tools (such as the Reproducing Kernel Hilbert Space, RKHS, framework) are more mature, and generalization error bounds are easier to derive. This is precisely the fundamental reason why the research team believes kernel methods deserve renewed consideration in Koopman embedding learning.

Bilevel Optimization: Bilevel optimization is a class of nested optimization problems in which the objective function of the outer optimization problem itself depends on the optimal solution of the inner optimization problem. The general form is $\min_{\theta} F(\theta, \phi^(\theta))$, where $\phi^(\theta) = \arg\min_{\phi} G(\theta, \phi)$. In EDMD-kDL, the outer optimization learns the parameters of the kernel dictionary (such as the hyperparameters of the kernel function or the weights of the dictionary basis), while the inner optimization solves the finite-dimensional approximation of the Koopman operator under that dictionary. The advantage of this structure is that it provides a principled mathematical framework for jointly optimizing the dictionary and the operator approximation, avoiding heuristic alternating update strategies. Bilevel optimization has been widely applied in meta-learning, hyperparameter optimization, and generative model training, and its gradients can be efficiently computed through the implicit function theorem or iterative differentiation techniques.

Scalability: A Key Breakthrough of Kernel Methods

A fatal weakness of standard kernel methods is that computational complexity grows dramatically with the size of the training set. EDMD-kDL addresses this issue by design: the size of the required kernel matrix depends on the number of collocation points, not the size of the training dataset.

This means the method naturally possesses scalability to large-scale datasets. For application scenarios with massive data volumes such as ocean temperature prediction, this property is particularly important.

Collocation Points: The collocation point method is a numerical technique for solving differential equations. Its core idea is to select a finite number of points within the solution domain and require the approximate solution to exactly satisfy the equation at these points, rather than everywhere in the entire domain. In the context of EDMD-kDL, collocation points serve as "anchors" for kernel dictionary learning—the construction of the kernel matrix only involves kernel function evaluations between these collocation points, so the matrix size is controlled by the number of collocation points and decoupled from the training set size. This stands in sharp contrast to traditional kernel methods: standard kernel ridge regression or kernel PCA requires computing an $N \times N$ kernel matrix (where $N$ is the number of samples), which becomes computationally and storage-wise infeasible when $N$ reaches the millions. By shifting complexity to a controllable number of collocation points, EDMD-kDL gains a key scalability advantage in large-scale data scenarios.

Experimental Validation: A Head-to-Head Comparison with ANN Methods

In a series of numerical experiments, the research team compared EDMD-kDL with current state-of-the-art ANN-based methods. The test scenarios included:

• Global sea surface temperature prediction: a typical high-dimensional nonlinear dynamical system modeling task
• Direct learning from video data: testing the method's modeling capability on raw perceptual data

Across all test settings, EDMD-kDL performed either on par with or better than ANN methods. This result is quite compelling—kernel methods have not been left behind by neural networks; on the contrary, they demonstrate competitiveness across multiple tasks.

Theoretical Significance and Application Prospects

This work reopens the space of possibilities for kernel methods in Koopman operator learning. Compared to the black-box learning process of neural networks, kernel methods provide a clearer path for theoretical analysis. For scientific research scenarios that require understanding the intrinsic mechanisms of models—such as climate modeling and fluid dynamics analysis—this interpretability may be more valuable than pure prediction accuracy.

At the same time, the introduction of the bilevel optimization framework provides a principled mathematical formulation for dictionary learning, rather than heuristic network architecture search. This points the way for future research: how to further enhance the practicality and applicability of the method while maintaining theoretical elegance.

One-line Summary

EDMD-kDL is the first to combine kernel methods with bilevel optimization to learn Koopman embeddings from data, achieving performance on par with ANN methods.

Paragraph Summary

EDMD-kDL proposes a new framework for Koopman embedding learning based on kernel methods, systematically combining kernel dictionary learning with bilevel optimization for the first time. Addressing the limitation that traditional EDMD requires a pre-specified dictionary, this method learns finite-dimensional Koopman embeddings directly from data through collocation points and a bilevel optimization framework, with the outer optimization learning the kernel dictionary and the inner optimization solving the Koopman approximation. Its key advantage is that the kernel matrix size depends only on the number of collocation points rather than the training set size, naturally providing scalability to large-scale data. In experiments on global sea surface temperature prediction and video data learning, EDMD-kDL achieves performance on par with or better than state-of-the-art ANN methods, while offering better theoretical interpretability and analytical tractability.