Why Large Language Models Fail at Tabular Prediction Compared to XGBoost

Introduction: LLMs Are Not Universal Prediction Tools

As Large Language Models (LLMs) demonstrate remarkable capabilities in natural language processing, code generation, and multimodal understanding, the industry has developed an optimistic expectation: as long as the model is large enough and the training data plentiful enough, it can handle virtually any prediction task. However, a counterintuitive reality is being confirmed by an increasing body of research and practice—when it comes to prediction tasks on structured tabular data, LLMs with hundreds of billions of parameters often lose to traditional gradient boosted trees (such as XGBoost and LightGBM) and other classical machine learning methods.

This topic recently sparked discussion on Hacker News. Although the discussion wasn't particularly heated, it touches on a long-overlooked technical blind spot: The architectural essence of language models makes them inherently unsuited for processing highly structured, numerically dense data like tables.

The Uniqueness of Tabular Data: Why It's So Difficult for LLMs

Structured and Numerically Dominant Data Forms

Tabular data is the most prevalent data form in the enterprise world: financial risk control, user profiling, sales forecasting, medical diagnostic records—the vast majority are stored in row-column structures. Its core characteristics include:

• Semantically heterogeneous columns: One column might be age (integer), another income (floating point), and yet another a category label (enumeration).
• No fixed ordering relationship between features: Unlike the sequential nature of text or the spatial continuity of images, the arrangement order of table columns typically carries no information.
• Sensitivity to numerical precision: Many predictions depend on precise numerical comparisons, such as "whether income exceeds a threshold."

These characteristics are fundamentally different from natural language. Language is a one-dimensional sequence with strong contextual dependencies and grammatical structure, while a table is an "unordered feature set" whose information is primarily embedded in numerical distributions and feature interactions.

Why Tree Models Like XGBoost Dominate Tabular Tasks

Gradient boosted trees, represented by XGBoost, recursively split on feature values and are naturally adept at capturing numerical thresholds, nonlinear interactions, and handling missing values. Their inductive bias matches tabular data perfectly—which is why tree models remain the undisputed champions for tabular tasks in data competitions like Kaggle.

The core idea of Gradient Boosted Decision Trees (GBDT) is to iteratively train multiple weak decision trees, where each new tree fits the residuals (prediction errors) of all preceding trees, and the final prediction is a weighted sum of all tree outputs. XGBoost builds upon this by introducing regularization terms, column sampling, and efficient split-point search algorithms (such as histogram acceleration), while LightGBM further adopts Leaf-wise growth strategy and GOSS (Gradient-based One-Side Sampling) to improve training speed. Tree models natively support mixed-type features (numerical + categorical), automatically handle missing values, and are insensitive to feature scale (no normalization needed)—properties that perfectly align with the heterogeneous nature of tabular data. A benchmark study by Grinsztajn et al. published at NeurIPS 2022 systematically demonstrated that GBDT-class models consistently outperform deep learning methods on medium-scale tabular datasets.

Three Fundamental Reasons Why LLMs Fail at Tabular Data

Tokenizers Destroy Numerical Semantics

The first step in LLM input processing is tokenization. When a number like 12345.67 is fed into the model, the tokenizer often splits it into multiple unrelated tokens (e.g., 123, 45, ., 67). This segmentation completely destroys the continuity and magnitude semantics of the number—what the model sees is no longer a "quantity" but a string of symbol fragments.

The tokenizers used by LLMs (such as BPE, WordPiece, SentencePiece, etc.) were originally designed for natural language. BPE (Byte Pair Encoding) iteratively merges high-frequency character pairs from the corpus to build a vocabulary, meaning the vocabulary construction is entirely based on text distribution rather than numerical semantics. For example, the number "12345" might be split into tokens "12" "34" "5", while "12346" might be split into "123" "46"—two numbers differing by only 1, but with completely different token representations, making it difficult for the model to infer their proximity. This representation prevents the model from establishing continuous numerical mappings and from reliably performing simple magnitude comparison operations.

This directly causes LLMs to be fragile in numerical comparison and arithmetic operations. For tabular predictions that heavily depend on numerical magnitude relationships, this is a fatal flaw.

Mismatch Between Transformer Architecture and Tabular Task Inductive Bias

The Transformer architecture was built for sequence modeling, with its self-attention mechanism excelling at capturing long-range dependencies between tokens. But tabular data doesn't need this capability—it needs efficient modeling of feature interactions and numerical splits. Serializing a table into text to feed an LLM essentially means using the wrong tool for the problem; the model must spend enormous parameter capacity to "rediscover" capabilities that tree models possess innately.

Inductive bias refers to the assumptions built into a model's architecture, determining what kinds of functions the model tends to learn with limited data. The Transformer's core inductive biases include: position-invariant attention weight computation (suitable for long-range dependencies in sequences) and sparse attention allocation through softmax. Tabular data requires fundamentally different inductive biases: axis-aligned splits (threshold cuts along a single feature), permutation invariance with respect to feature ordering, and sensitivity to local numerical intervals. The Transformer's attention mechanism tends to learn smooth weighted combinations between features, rather than step-like piecewise constant functions (which is precisely what decision trees excel at). Experiments show that Transformers require extremely large amounts of data to approximate the piecewise decision capability that tree models possess innately.

Severe Imbalance Between Data Efficiency and Inference Cost

Tree models can achieve good results with just a few thousand samples, while LLMs—even when processing tables through fine-tuning or in-context learning (ICL)—require more samples, higher inference costs, and still often fall short of baselines. In production environments, this "high cost, low return" combination lacks appeal.

Specifically, a single XGBoost training run takes only seconds to minutes on a regular CPU, with per-sample inference latency at the microsecond level; serializing a table and feeding it to an LLM for inference can consume hundreds of milliseconds per sample while occupying expensive GPU resources. When facing batch prediction scenarios with millions of samples (such as daily credit card fraud detection), LLM inference costs can be three to four orders of magnitude higher than tree models—completely unacceptable commercially. Additionally, the in-context learning approach is limited by the LLM's context window length, making it impossible to include an entire large-scale training set in the prompt.

Improvement Directions: Bridging the Gap Between LLMs and Tabular Prediction

Specialized Numerical Encoding Schemes

Academia is already exploring improvement directions. For example, designing specialized encoding methods for numbers (rather than relying on general-purpose tokenizers), or introducing continuous value embedding layers that allow models to understand the true magnitude of numbers. Such methods have narrowed the gap on some benchmarks but have not yet displaced tree models' leading position.

Representative approaches include: converting numbers to scientific notation (allowing the model to separately perceive magnitude and significant digits), using methods like xVal to multiply numerical scalars directly into embedding vectors (bypassing discrete tokenization), and quantile encoding (mapping numbers to the percentile interval they occupy in the training set). Google Research has also explored schemes that assign independent tokens to each digit with attached place-value information. The common goal of these methods is to enable neural networks to process numbers as if perceiving "magnitude relationships" rather than treating them as symbol sequences.

Tabular-Specific Foundation Models Like TabPFN

Another path is designing foundation models specifically for tabular data, such as TabPFN. These borrow some ideas from Transformers but deeply customize the architecture for tabular characteristics, rather than directly applying language models. These "tabular-specific foundation models" represent a more pragmatic direction—not making LLMs learn tables, but designing appropriate models for tables.

TabPFN (Tabular Prior-Data Fitted Network), proposed by Hollmann et al. in 2023, has a core idea of pretraining a Transformer on a large number of synthetic tabular datasets, teaching it the meta-task of "how to do tabular prediction" (meta-learning). At inference time, it takes the training set as context input and directly outputs predictions for test samples without gradient updates, similar to a neural network that has learned Bayesian inference. TabPFN has already shown performance competitive with or even surpassing XGBoost on small-scale datasets (<1000 samples, <100 features). Additionally, architectures like FT-Transformer (which independently embeds each feature before feeding into a Transformer) and SAINT (combining row attention and column attention in a dual attention mechanism) are all customized designs targeting tabular data characteristics, representing the correct approach of "designing architectures for data forms" rather than "forcing data to adapt to architectures."

The Proper Role of LLMs in Tabular Scenarios

This doesn't mean LLMs have no value in tabular scenarios. They're better suited for auxiliary roles: automatically generating feature engineering code, explaining prediction results, processing text fields within tables, and assisting with data cleaning. Combining LLMs' language understanding capabilities with tree models' predictive power is often more effective than relying on either alone.

For example, Microsoft's FLAME framework and multiple open-source projects have demonstrated this combined paradigm: LLMs handle understanding business context, suggesting meaningful feature combinations (such as converting "date of birth" to "age" then to "age group"), automatically writing data preprocessing pipeline code, and even generating human-readable explanation reports for model predictions. The actual predictive modeling is still handled by tree models like XGBoost/LightGBM. This collaborative pattern of "LLM as data scientist copilot + tree model as predictor" leverages LLMs' semantic understanding advantages while preserving tree models' precision and efficiency in numerical prediction—representing the most pragmatic best practice in industry today.

Conclusion: Choosing the Right Tool for the Task

LLMs' "failure" at tabular prediction isn't due to insufficient model capability, but is a classic case of architecture-task mismatch. It reminds us that even in an era of rapidly advancing AI technology, we must maintain engineering sobriety: no single model is universal.

For structured tabular data, mature gradient boosted trees remain the first choice; for scenarios requiring semantic understanding, LLMs can truly shine. Truly efficient systems often come from precisely grasping and combining the strengths of different tools, rather than blindly chasing parameter scale. This also echoes a classic but often forgotten principle in machine learning—the No Free Lunch Theorem: there is no algorithm that is optimal across all problems; every method's advantages are built upon specific assumptions. Revisiting this principle amid the LLM hype holds significant practical value for avoiding bandwagon effects in technology selection.