How Well Do PFNs Approximate Bayesian Inference? An Empirical Analysis of Amortization Error

Understanding PFNs Through the Lens of Amortized Learning

Prior-data Fitted Networks (PFNs) represent a significant innovation in tabular data modeling. The core idea is straightforward: by pretraining on large amounts of synthetic prior data, a neural network learns to directly approximate the Bayesian Posterior Predictive Distribution (PPD). This approach "amortizes" the repeated Bayesian computations that would otherwise be needed at inference time into a one-time pretraining process — in other words, the model learns a direct mapping from data to predictions, without requiring explicit posterior sampling at deployment.

This is an elegant idea, but a critical question has long been overlooked: to what extent do PFNs actually approximate the target posterior predictive distribution? This study notes that empirical results on PFN approximation quality are "surprisingly thin" — and that is precisely the gap it aims to fill.

The Bayesian PPD is the core output of Bayesian inference: given observed data (context), it describes the full probability distribution over labels for new samples, rather than a single-point prediction. Traditional methods require integrating over the posterior distribution of model parameters — a computationally expensive process that typically demands Markov Chain Monte Carlo (MCMC) sampling or variational inference. The concept of "amortization" is borrowed from amortized variational inference: rather than performing inference from scratch on each new dataset, a network is trained on many examples to directly learn the inference process itself, "prepaying" inference costs at training time. PFNs take this idea to its logical extreme — the Transformer architecture naturally handles variable-length contexts, allowing the network to take a training set directly as input at inference time and output a predictive distribution, without any parameter updates.

Decomposing Deployment Error into Three Components

This study adopts an Empirical Risk Minimization (ERM) perspective to systematically decompose the errors a PFN produces during deployment into three sources:

• Prior error: the gap between the synthetic prior and the true data distribution;
• Context error: information insufficiency caused by limited context samples;
• Amortization error: the approximation bias introduced by the network itself when approximating the target PPD.

The value of this decomposition lies in transforming a vague "bad predictions" problem into components that can be measured and attributed separately. The study focuses on amortization error, measuring PFN outputs against a reference PPD under a realistic prior setting. This experimental design makes it possible to quantify exactly where PFNs go wrong, rather than relying on intuition.

The ERM framework formulates learning as minimizing the expected loss over finite samples to approximate a true target function. When applied to PFN error analysis, the three error types correspond to three different approximation gaps: prior error resembles "model bias" in traditional machine learning — systematic deviation between the synthetic data generation process and real-world distributions; context error corresponds to variance in statistical estimation — finite training samples cannot fully characterize the data-generating process; amortization error is unique to this paradigm — even with a perfect prior and infinite samples, a neural network's finite capacity and the stochasticity of training still introduce additional bias. This orthogonal decomposition transforms the vague phenomenon of "a model performing poorly on some dataset" into something diagnosable and attributable, providing a theoretical handle for targeted improvement.

Generally Good Performance, but Stubborn Residual Error

The empirical results deliver a relatively optimistic baseline conclusion: PFNs can generally reproduce the target posterior predictive distribution well. In most standard scenarios, amortized learning does work — the network captures the essential behavior of Bayesian inference.

However, the study also identifies a concerning phenomenon: some amortization error stubbornly persists, especially in rare scenarios within the pretraining prior distribution. More critically, this residual error cannot be eliminated simply by scaling up the model — even with a larger model, the approximation bias in rare settings remains.

The outward manifestation of this error is prediction disagreement between independently pretrained PFNs. When two models trained with the same method but different random seeds give inconsistent predictions for the same input, it signals additional uncertainty at inference time about "which predictor is correct." This uncertainty does not stem from the data itself, but from the imperfection of the amortization process.

Bayesian Model Averaging as a Remedy

To address this residual uncertainty, the study proposes and validates a practical remedy: Bayesian Model Averaging (BMA).

Intuitively, since independently trained PFNs diverge, averaging their predictions in a principled way should cancel out their individual random amortization biases. The study examines several types of Bayesian model averaging approaches and finds that these methods consistently reduce inference-time uncertainty and improve prediction quality.

Particularly noteworthy is that this improvement holds even for the largest-scale TabICLv2 model — even a full-sized, powerful model still benefits from model averaging. This corroborates the earlier finding: amortization error is a structural problem, not one that can be solved simply by adding more parameters, and ensemble-style averaging strategies provide a more effective path forward.

The theoretical basis for BMA is that when multiple plausible models exist, the truly Bayesian treatment is not to select the single "best" one, but to take a weighted average of all models' predictions according to their posterior probabilities. In the deep learning context, multiple models trained with different random initializations can be viewed as samples from model space — each embodying different implicit assumptions about "which inference procedure is more appropriate." For PFNs, each independent training run amounts to a different random realization of the amortization error; averaging these realizations cancels out random biases, analogous to variance reduction in ensemble learning. The distinction from ordinary ensemble methods is that BMA carries a clear Bayesian interpretation: it is not merely an engineering trick, but a more complete probabilistic treatment of model uncertainty. The fact that improvement remains possible even on TabICLv2 — currently one of the largest tabular in-context learning foundation models — indicates that residual amortization error is systemic rather than a capacity bottleneck.

Implications for Practice and Research

This work delivers practical value for both practitioners and researchers working with tabular foundation models.

For practitioners, one directly actionable takeaway is: when relying on PFNs for high-stakes or rare-scenario predictions, do not blindly trust the output of a single pretrained model. Instead, consider using Bayesian averaging across multiple independently trained models to reduce uncertainty. The disagreement between independent models can itself serve as a signal for identifying cases where "the model may be unreliable."

For researchers, the three-way error decomposition framework (prior, context, amortization) provides a clear diagnostic tool for identifying PFN performance bottlenecks. It reminds the community that evaluating PFNs cannot rely solely on average metrics — equal attention must be paid to approximation degradation in distribution tails and rare cases, which are typically the weakest points of amortized learning.

Overall, the paradigm of "amortizing Bayesian inference into network weights" via PFNs is both viable and impressive in practice — but it is not a perfect substitute for true Bayesian computation. Recognizing the boundaries of amortization error, and compensating with strategies like model averaging, is a critical step toward making these methods reliably deployable.