[KongchangAI]
· 2 min read· 1,451 words

Parameter-Free Tabular Foundation Models Are Challenging Gaussian Processes' Dominance in Materials Discovery

Parameter-Free Tabular Foundation Models Are Challenging Gaussian Processes' Dominance in Materials Discovery

Tabular foundation models can serve as Bayesian optimization surrogates without kernel selection, rivaling Gaussian processes on materials discovery benchmarks.

This study explores the feasibility of replacing Gaussian processes with tabular foundation models (TFMs) as surrogate models for Bayesian optimization in materials discovery. Although Gaussian processes are the default choice, they rely on kernel engineering and have limited ability to model complex structure–property relationships. The study evaluates two TFMs, TabPFN and TabICL, against baselines including Gaussian processes, random forests, and CatBoost on two materials science benchmarks. Under an "out-of-the-box" constraint, TFMs showed highly competitive performance without task-specific training or hyperparameter tuning. The article analyzes the significance of this result: lowering the barrier to use and potentially offering stronger modeling capability, though not comprehensively surpassing GPs. It also notes dimensions that require caution, such as limited benchmark coverage, uncertainty quantification quality, and inference cost, and argues that the results support further development of TFMs as surrogate models for active learning.

When Bayesian Optimization Meets Materials Discovery: Gaussian Processes' "Default Option" Status Is Being Shaken

In the active learning loop of materials science, Bayesian optimization has long played a central role. It uses surrogate models to predict expensive experimental or simulation results and to guide the direction of the next round of experiments. Gaussian Processes (GPs) are almost the default choice in this workflow—they provide uncertainty estimates, have a mature mathematical framework, and have been validated as effective in a large body of literature.

But Gaussian processes are not without cost. They require users to choose an appropriate kernel for the specific task, and kernel selection often depends on domain expertise, while the optimal kernel can differ dramatically across material systems. In addition, when faced with complex structure–property relationships, GP modeling capability can sometimes be insufficient.

A recent study on arXiv proposes a noteworthy alternative: directly using Tabular Foundation Models (TFMs) as surrogate models for Bayesian optimization, with no kernel selection and no additional training for active learning tasks.

rss source: Out-of-the-Box Tabular Foundation Models for Bayesian Optimization in Materials Discovery

What Are Tabular Foundation Models?

Tabular foundation models are a class of models that have emerged in recent years for structured data prediction tasks. Unlike traditional tree models or Gaussian processes, TFMs are pretrained on large-scale tabular data and can directly make predictions for new tabular tasks without redesigning features or tuning hyperparameters for each task.

The study evaluates two representative TFMs: TabPFN and TabICL. TabPFN is based on the Transformer architecture and gains generalization ability through pretraining on large-scale synthetic tabular data; TabICL is another type of foundation model designed for tabular data. Their common feature is being "out of the box"—given training data, they directly output predictions without kernel engineering or heavy hyperparameter search.

This stands in sharp contrast to how Gaussian processes are used. GPs require selecting a kernel type (such as RBF, Matern, etc.) and tuning hyperparameters such as length scale, whereas TFMs internalize these decisions into the pretraining stage.

The core idea of TabPFN comes from Prior-Data Fitted Networks. Rather than learning patterns of a specific dataset during training, it learns a "meta-mapping" from training sets to prediction functions by pretraining on massive synthetic tabular datasets. At inference time, given a small training set and test points, TabPFN can output a predictive distribution through a single forward pass without gradient updates. This design gives it inherent uncertainty estimation capability and insensitivity to hyperparameters. TabICL adopts a different architectural approach, usually combining in-context learning with tabular-specific inductive biases to handle larger-scale or more complex tabular tasks. Together, the two represent the extension of the "foundation model" paradigm into the tabular data domain, and their key breakthrough lies in transforming the model selection and tuning process that traditionally required manual design into a one-time investment during pretraining.

Experimental Design: A Head-to-Head Comparison on Two Materials Science Benchmarks

The researchers compared TFMs with a broad class of baseline models on two standard materials science benchmark datasets. The baselines cover mainstream methods such as Gaussian processes, random forests, and CatBoost, forming a fairly comprehensive comparison field.

The core question of the evaluation is: without task-specific training for active learning and without GP kernel selection, can TFMs serve as competitive surrogate models?

The key to the experimental design lies in the "out-of-the-box" constraint. This means TFMs were not additionally fine-tuned, nor were they adapted in any way for materials discovery tasks. If they can still achieve performance comparable to carefully tuned baselines under these conditions, it shows that TFMs have the potential to serve as general-purpose surrogate models.

Interpreting the Results: Where Does TFMs' Competitiveness Come From?

The core finding of the study is that on both benchmarks, TFMs showed highly competitive performance without task-specific training or GP kernel selection.

The significance of this result can be understood on several levels.

First, it lowers the barrier to use. When materials scientists use Bayesian optimization, they no longer need to spend substantial effort on kernel selection and hyperparameter tuning. TFMs move this work upstream to the pretraining stage, and users only need to provide data to obtain predictions.

Second, differences in modeling capability deserve attention. The paper notes that Gaussian processes "may lack the ability to model complex relationships." Structure–property relationships in materials data are often highly nonlinear and involve multiscale interactions, and the representational capability that TFMs acquire from large-scale pretraining may offer advantages in this regard.

Third, this is not a rejection of Gaussian processes. The researchers describe TFMs as "highly competitive" rather than comprehensively superior. GP's theoretical guarantees for uncertainty quantification and mature toolchain remain advantages. TFMs are positioned as a strong alternative, especially in scenarios where users are unwilling or unable to invest in kernel engineering.

In Bayesian optimization, the acquisition function is responsible for balancing exploration and exploitation, with common choices including expected improvement (EI), upper confidence bound (UCB), and Thompson sampling. These functions depend directly on the predictive mean and uncertainty output by the surrogate model. Gaussian processes are naturally suited to this because of their analytical form of uncertainty quantification, whereas TFMs, as neural-network-like models, usually derive uncertainty estimates from parameterized outputs of predictive distributions or ensemble methods. If TFMs' uncertainty is poorly calibrated, acquisition functions may favor excessive exploration or premature convergence. Therefore, evaluating the practical value of TFMs in Bayesian optimization cannot rely solely on predictive accuracy; it is also necessary to test whether their uncertainty can effectively guide experimental selection in the active learning loop. This is also one of the directions the paper identifies as needing further development.

Potential Impact on Materials Discovery Workflows

If TFMs' competitiveness is validated on more benchmarks, the practice of active learning in materials discovery may change.

The current typical workflow is: choose a GP kernel → train a surrogate model → select the next round of experiments via an acquisition function → update data and repeat. Kernel selection is often fixed early in a project, and later adjustment is costly. The introduction of TFMs can simplify this workflow and make surrogate model selection no longer a bottleneck.

The paper's conclusion also points to future directions: the results "motivate developing TFMs further as surrogates for active learning in materials discovery"—that is, it is worth further developing TFMs to make them surrogate models for active learning in materials discovery. This implies that current TFMs still have room for improvement, especially in uncertainty estimation and acquisition function adaptation in active learning settings.

Points Requiring Caution

The scope of this study's evaluation is two standard benchmarks. The diversity of materials science means that benchmarks cover only a limited set of systems, and the performance of TFMs on a broader range of material classes still needs verification. In addition, the core of Bayesian optimization lies not only in predictive accuracy but also in the quality of uncertainty estimation—which directly determines the effectiveness of acquisition functions. The paper's abstract does not elaborate in detail on TFMs' performance in uncertainty quantification, and this is where readers need to pay attention to the details in the original text.

Another practical issue is inference cost. TFMs are based on the Transformer architecture, and how their inference overhead on large datasets compares with GPs needs to be judged in light of specific scales. The paper's abstract does not address this dimension, but in practical materials discovery workflows, surrogate models need to be updated and used for prediction quickly in each iteration, so cost factors cannot be ignored.

Benchmark datasets in materials science are usually limited in size (tens to hundreds of samples), which differs in distribution from the large-scale synthetic data that TFMs are exposed to during pretraining. Although TFMs are designed to be friendly to small-sample tasks, the heterogeneity of materials data—such as the high diversity of crystal structures, composition spaces, and synthesis conditions—may exceed the pretraining distribution. In addition, each iteration of active learning adds new data points, and whether TFMs can maintain stable predictive performance on incremental data without retraining or fine-tuning is a key issue in practical deployment. Existing benchmark tests mostly use fixed datasets for offline evaluation, which still differs from the dynamic data streams in real closed-loop experiments, so conclusions should be extrapolated with caution.

Summary

As out-of-the-box surrogate models for Bayesian optimization, tabular foundation models have shown highly competitive performance against baselines such as Gaussian processes, random forests, and CatBoost on two materials science benchmarks. The core value of this result is that it provides a path that bypasses kernel engineering and hyperparameter tuning while potentially offering stronger modeling of complex relationships. For practitioners of active learning in materials discovery, TFMs are worth including in the candidate list of surrogate models.

#tabular foundation models#Bayesian optimization#materials discovery#TabPFN#TabICL#Gaussian processes
Share:

Related articles