DuckDB and Arrow Dominate AI Data Preparation—Why Do They Hit a Wall in Inference Serving?

TL;DR — Columnar engines (such as Arrow, DuckDB, Polars, and GPU DataFrames) excel at scanning large numbers of rows and computing aggregates, making them ideal for building training sets; but they perform terribly when it comes to single-row inference lookups. The core value of Arrow in AI systems is not query speed, but rather enabling zero-copy data exchange between the offline columnar world and the online row-based world. Teams that try to make one engine handle both will pay the price in p99 latency.

Every few months, a new benchmark shows DuckDB, Polars, or GPU DataFrame engines crushing the competition on TPC-H-style workloads. These numbers are real, but the conclusion people draw from them—that columnar engines have become the end-to-end default backbone of AI data infrastructure—is exactly wrong.

Columnar engines excel at specific tasks: scanning large numbers of rows to compute a small number of aggregates. However, they have structural deficiencies in what most AI inference paths actually need—fast retrieval of single rows under high concurrency. These are not the same problem, yet the industry keeps building systems that quietly assume they are interchangeable.

What Exactly Does Columnar Storage Optimize?

Columnar storage exists because analytical queries typically touch only a few columns while scanning massive numbers of rows. Storing by column means reading only the bytes you need, compressing similar values together, and enabling vectorized operations across contiguous memory with almost no branch prediction. This is precisely the core bet shared by Arrow's columnar memory format, DuckDB's execution engine, and GPU DataFrame libraries: amortizing fixed overhead across millions of values.

For building training sets, this bet is exactly right. You need to join event fact tables with user attribute dimension tables, compute rolling aggregates, filter, deduplicate—all of which are scan-reduce operations over large batches. Columnar engines perform these tasks an order of magnitude faster than row-based databases, and GPU-accelerated columnar execution further improves performance by parallelizing the same vectorized operations across thousands of cores.

However, none of this has anything to do with what happens at inference time.

Zero-copy is the key concept for understanding Arrow's value. In traditional data exchange, one engine writes results to a file or memory buffer, and the next engine reads and parses them into its own internal format—involving at least one serialization and one deserialization, with memory allocation and data copying. Zero-copy means multiple processes or libraries share the same physical memory, each reading directly through a view in its expected layout, with no data movement. Arrow defines a language-agnostic memory layout specification (columnar, fixed-length offsets, buffer alignment) so that Pandas, Polars, DuckDB, PyTorch, and others can pass data pointers rather than data itself, as long as they follow the spec. The cost is that both sides must accept the same layout constraints; once types need to change or data needs to be reordered, the zero-copy advantage disappears.

Vectorized execution means the query engine processes data not row by row, but in batches (typically 1024 to 8192 rows), calling SIMD instructions or tight loops over entire columns, amortizing loop overhead and branch checks across the whole batch. This is exactly why columnar engines are an order of magnitude faster in scan-reduce scenarios, and it also explains why this advantage completely disappears for single-row lookups: when batch size is one, there is nothing to amortize, leaving only the overhead of reconstructing column by column.

Where Does the Model Break Down?

An inference request—whether a chat turn, an agent step, or a ranking call—needs a small number of specific rows: the current feature vector for that user, the embedding for that document, the running state for that session. This is a point lookup, not a scan. In columnar storage, reconstructing a single logical row means touching N separate column chunks, each potentially compressed, potentially located in different pages or files. Whether you need one row or a million, you pay this reconstruction cost. For a million rows, it can be masked by vectorized throughput; but for one row, under p99 latency and concurrent load, it is pure latency with no amortization to hide it.

This is why no one serves a production recommendation feature store by rescanning Parquet files on every request. This is also why vector databases don't store vectors in general-purpose columnar tables and brute-force scan them—they build specialized HNSW or IVF indexes precisely because the access pattern is "immediately find the nearest neighbors to the query vector," which is the exact opposite of what columnar layouts are good at. The vector database field has learned this lesson, but the broader AI data stack has not yet internalized it.

The same mistake appears in agent systems. An agent step that needs to read conversation state, tool outputs, and long-term memory fragments performs dozens of small, latency-sensitive point reads per turn. If the underlying engine is a columnar analytical engine, you build a system that is extremely fast at answering "how many agent runs happened last week" but painfully slow at answering "what did this particular agent remember five turns ago."

HNSW (Hierarchical Navigable Small World) and IVF (Inverted File Index) are two mainstream types of approximate nearest neighbor indexes in vector databases. HNSW builds a multi-layer graph structure; queries greedily descend from a sparse top layer, visiting only a few neighbor nodes per layer, approaching the nearest neighbor with logarithmic complexity. IVF first uses k-means to cluster vectors into groups, and at query time scans only the clusters closest to the query vector, trading a small amount of recall for speed. What they share is that they are both designed for the point lookup access pattern—"given a query vector, immediately find the nearest K"—rather than sequential scanning of all vectors. If you forcibly store vectors in a general-purpose columnar table and brute-force compute distances, every query must read and decode all vector columns—precisely the "completely opposite access pattern" the article describes. The index structures of vector databases are themselves an engineering response to the fact that columnar layouts are unsuitable for online retrieval.

Arrow's Real Contribution Is Not the Query Engine

This is where Arrow truly delivers value, and it is the part most benchmarks ignore. Arrow's contribution to AI infrastructure is providing a shared, zero-copy columnar memory layout that allows multiple engines to read and write without serializing between each other. Pandas, Polars, DuckDB, Spark, PyTorch data loaders, GPU DataFrames—they can all pass the same block of memory without paying a conversion tax at every boundary.

This matters enormously for the offline side of AI pipelines, where data passes through five or six tools from raw logs to training batches. Every serialization boundary eliminated saves real wall-clock time and infrastructure cost, independent of any single engine's query planner speed. The headline-grabbing part of the Arrow ecosystem is "look how fast this SQL query runs," while the quietly critical part is "look how many copy operations between tools that nobody thought to consolidate this eliminates."

Conflating these two contributions is a mistake. Teams read "Arrow is fast" and "DuckDB is fast" and conclude that the columnar stack should extend all the way to the serving boundary. It should not, and forcing it to do so produces inference paths that are fast in load tests with batch size 10,000 but collapse under real traffic (batch size one, repeated continuously, latency budgets in milliseconds).

The classic dual-storage pattern of a feature store deserves elaboration. The offline store is typically a large-volume columnar warehouse (Parquet/Delta/Iceberg with Spark or DuckDB compute), used for batch computing features, backfilling history, and ensuring consistent feature definitions between training and inference; the online store is a low-latency key-value store (such as Redis, DynamoDB, or a custom in-memory KV), holding the latest feature values keyed by entity for millisecond-level reads at inference time. Between the two, a materialization pipeline is needed: feature batches computed offline are written to the online store via the Arrow exchange format, avoiding the serialization overhead of text formats like JSON. The core problem this pattern solves is training-serving skew: if offline and online use different computation logic, the model's online performance will diverge from its evaluation results. Thus dual storage is not a transitional solution but a structural choice that balances throughput and latency.

Design for Two Speeds, Not One

The architecture that actually works treats this as two different problems and connects them at a well-defined seam. On the offline side, use columnar and GPU-accelerated stacks for what they do best: building and refreshing training datasets, computing batch feature transformations, backfilling, and running analytical evaluations over logged inference traffic. This is where Arrow-based engines and GPU DataFrames deliver real, repeatable gains, because the workloads truly are scan-reduce over large data volumes.

On the online side, use row-based storage, key-value layers, or purpose-built indexes—anything that can deliver single-digit millisecond point lookups at target concurrency. The feature store pattern—pairing an offline columnar warehouse with a low-latency online store—is not a legacy artifact waiting to be modernized away by faster columnar engines; it is the correct shape of the problem, and GPU acceleration on the columnar side does not change which side of the seam a given workload belongs to.

The seam itself is where Arrow belongs—as an exchange format that lets the offline columnar pipeline pass freshly computed feature batches to the process populating the online store, without a serialization detour through JSON or custom binary formats. This is a real and meaningful win, just a different one from "look how fast this query runs."

One-line Summary

Columnar engines excel at batch scan-aggregation but are ill-suited for single-row inference lookups; Arrow's true value is zero-copy exchange, not query speed.

Paragraph Summary

The article refutes the popular notion that columnar engines should become the end-to-end default backbone of AI data infrastructure. Columnar storage is optimized for scanning massive rows and computing a small number of aggregates, giving it a clear advantage when building training sets; but inference requests are fundamentally single-row point lookups under high concurrency, requiring traversal of N separate column chunks and column-by-column reconstruction, where amortization advantages vanish and tail latency cannot be hidden. Vector databases building specialized HNSW/IVF indexes and agent systems performing dozens of point reads per turn both confirm this. The author points out that Arrow's core contribution is a shared zero-copy memory layout that eliminates serialization boundaries between tools in offline pipelines—not query speed. The correct architecture should separate offline columnar processing from online row-based storage, with Arrow serving as the exchange format at the seam between the two.