Tabular Foundation Model
A tabular foundation model is a class of model pretrained on large, heterogeneous tabular datasets that can learn relationships among columns, cell semantics, and table structure, enabling prediction without manual feature engineering. Representative works include Kumo and Nori.
Timeline (last 90 days)
Tabular foundation models are pretrained on millions of different tabular prediction tasks; when facing a new dataset, they learn patterns directly from examples in the table through in-context learning (ICL) without rebuilding the model.
When actually deploying tabular foundation models, the capacity of the KV cache and multi-tenant isolation must be considered; different datasets correspond to different caches, which need to be managed by session or task.
In tabular foundation models, KV cache reuse can separate the initial loading of examples from subsequent queries, thereby reducing single-prediction latency to sub-second levels.
The cost of ICL is that every prediction must carry examples, and the longer the context, the higher the inference cost; the selection and ordering of examples affect performance; scenarios with extremely large data volumes or very high feature dimensions still require sampling and retrieval strategies.
Competition in tabular foundation models is shifting from model architecture to pretraining data recipes, and the scale and diversity of synthetic data have become the core variables that create gaps.
Tabular foundation models typically need to serialize a row of samples into a token sequence and adopt column-aware encoding to handle heterogeneous schemas.
Benchmark datasets in materials science are typically limited in scale (tens to hundreds of samples) and differ in distribution from the large-scale synthetic data that TFMs encounter during pretraining; the heterogeneity of materials data may fall outside the pretraining distribution range.
TFMs are based on the Transformer architecture, and how their inference overhead on large datasets compares with GPs needs to be judged in light of the specific scale; the paper's abstract does not address the inference cost dimension, while in actual materials discovery workflows the surrogate model needs to be rapidly updated and make predictions in each iteration round.
Acquisition functions in Bayesian optimization (such as EI, UCB, and Thompson sampling) directly rely on the predicted mean and uncertainty output by the surrogate model; if the uncertainty calibration of TFMs is poor, the acquisition function may bias toward over-exploration or premature convergence.
A recent study from arXiv proposes using tabular foundation models (TFMs) as surrogate models for Bayesian optimization, requiring no kernel function selection and no additional training for active learning tasks.
9 more timeline events
All Facts (19)
When actually deploying tabular foundation models, the capacity of the KV cache and multi-tenant isolation must be considered; different datasets correspond to different caches, which need to be managed by session or task.
50%UnverifiedIn tabular foundation models, KV cache reuse can separate the initial loading of examples from subsequent queries, thereby reducing single-prediction latency to sub-second levels.
50%UnverifiedThe cost of ICL is that every prediction must carry examples, and the longer the context, the higher the inference cost; the selection and ordering of examples affect performance; scenarios with extremely large data volumes or very high feature dimensions still require sampling and retrieval strategies.
50%UnverifiedTabular foundation models are pretrained on millions of different tabular prediction tasks; when facing a new dataset, they learn patterns directly from examples in the table through in-context learning (ICL) without rebuilding the model.
50%UnverifiedTabular foundation models typically need to serialize a row of samples into a token sequence and adopt column-aware encoding to handle heterogeneous schemas.
50%UnverifiedCompetition in tabular foundation models is shifting from model architecture to pretraining data recipes, and the scale and diversity of synthetic data have become the core variables that create gaps.
50%UnverifiedAcquisition functions in Bayesian optimization (such as EI, UCB, and Thompson sampling) directly rely on the predicted mean and uncertainty output by the surrogate model; if the uncertainty calibration of TFMs is poor, the acquisition function may bias toward over-exploration or premature convergence.
50%UnverifiedThe researchers compared TFMs with baseline models such as Gaussian processes, random forests, and CatBoost on two standard materials science benchmark datasets.
50%UnverifiedThe study's core finding is that on both benchmarks, TFMs demonstrated highly competitive performance without requiring task-specific training or GP kernel function selection.
50%UnverifiedThe researchers' wording is that TFMs are "highly competitive" rather than comprehensively surpassing Gaussian processes; GP's theoretical guarantees for uncertainty quantification and mature toolchain remain its advantages.
50%UnverifiedBenchmark datasets in materials science are typically limited in scale (tens to hundreds of samples) and differ in distribution from the large-scale synthetic data that TFMs encounter during pretraining; the heterogeneity of materials data may fall outside the pretraining distribution range.
50%UnverifiedTFMs are based on the Transformer architecture, and how their inference overhead on large datasets compares with GPs needs to be judged in light of the specific scale; the paper's abstract does not address the inference cost dimension, while in actual materials discovery workflows the surrogate model needs to be rapidly updated and make predictions in each iteration round.
50%UnverifiedTabular foundation models are sensitive to column semantics and data types, and improper handling of time alignment can easily introduce leakage; moreover, their inference cost and interpretability are often less transparent than those of gradient-boosted trees.
50%UnverifiedTabular foundation models are a class of models pretrained on large heterogeneous tabular datasets, with representative work including Kumo and Nori; their typical usage is to organize the samples to be predicted into a single table, and the model directly infers based on the tabular priors obtained from pretraining.
50%UnverifiedData distribution shift may cause zero-training prediction to underperform models trained specifically for the task, especially when business data has strong domain characteristics (such as medical coding or rare events in financial risk control).
50%UnverifiedWhen current mainstream tabular foundation models (TFMs) are used directly for survival analysis tasks, none of them can consistently beat the Cox model.
50%UnverifiedA recent study from arXiv proposes using tabular foundation models (TFMs) as surrogate models for Bayesian optimization, requiring no kernel function selection and no additional training for active learning tasks.
50%UnverifiedThe paper's conclusion states that the results "motivate developing TFMs further as surrogates for active learning in materials discovery," implying that current TFMs still have room for improvement in uncertainty estimation and acquisition function adaptation in active learning scenarios.
50%UnverifiedFor small- to medium-scale datasets, pre-training-based tabular foundation models have the potential to match or even surpass carefully tuned tree models at very low hyperparameter-tuning cost.
50%