From MMAP to io_uring: An Optimization Experiment That Made Things Slower

A Counter-Intuitive Performance Experiment

In the world of system performance optimization, we're constantly surrounded by claims that "newer technology is faster." io_uring, Linux's most talked-about asynchronous I/O interface in recent years, is widely believed to dramatically improve performance for I/O-intensive applications. Yet a team building a query engine in Rust shared a counter-intuitive finding: when they replaced their traditional MMAP (memory-mapped file) approach with io_uring, their system actually got slower.

This case is worth examining closely precisely because it challenges the engineering community's tendency toward "technology worship" — not every cutting-edge technology delivers performance gains in every scenario. Technology selection is always an art of context and trade-offs.

Understanding the Fundamental Differences Between the Two I/O Approaches

MMAP: Accessing Files Like Memory

MMAP (Memory-Mapped File) is a technique that maps files directly into a process's virtual address space. For a query engine, MMAP's biggest advantage is that it hands off the complexity of I/O to the operating system kernel.

When the query engine needs to read data, it simply accesses the mapped region like ordinary memory. The kernel uses page fault mechanisms to load disk data into the page cache on demand. The benefits are clear:

• Zero-copy: Data doesn't need to be copied back and forth between kernel buffers and user buffers
• Automatic caching: The OS page cache automatically manages hot data
• Simpler programming: Developers don't need to manually manage the lifecycle of I/O requests

io_uring: The Face of Modern Async I/O

io_uring is an asynchronous I/O framework introduced in Linux 5.1. Through two ring buffers — a submission queue (SQ) and a completion queue (CQ) — it enables efficient batch I/O communication between user space and kernel space. It was designed to address the limitations of the traditional AIO interface, and in theory can handle massive concurrent I/O requests with minimal syscall overhead.

In high-concurrency, high-throughput storage scenarios, io_uring often delivers significant performance advantages, which is why it has earned such a strong reputation.

Why the Replacement Made Things Slower

While the original report offered limited technical detail, from a systems engineering perspective we can reasonably infer several key reasons.

Discarding the Hidden Advantage of the Page Cache

When using MMAP, the query engine is effectively standing on the shoulders of a giant — the OS page cache. Query engine workloads typically involve repeated access to the same hot data. MMAP lets this data naturally reside in the page cache, making subsequent accesses nearly pure memory operations — extremely fast.

After switching to io_uring, if the team used O_DIRECT to bypass the page cache, they gave up this free caching acceleration. Unless the application layer implemented an equally efficient caching mechanism, every read would need to reach all the way down to the storage device.

A Mismatch Between Syscall Batching and Access Patterns

io_uring's performance advantage is built on a model of batch submission and asynchronous completion. It excels when handling hundreds or thousands of concurrent I/O requests simultaneously, amortizing overhead through a single batch syscall.

However, if the query engine's access pattern is fundamentally random and small-batch in nature, io_uring's batching advantage never gets a chance to shine. Instead, it introduces additional overhead from queue management, request construction, and completion event polling.

The Extra Cost of Data Copies and Memory Management

MMAP's zero-copy property means data can be used directly by query logic once loaded. With io_uring, you typically need to pre-allocate read buffers, and after data is read from the device into those buffers, it may require additional processing before upper-layer logic can consume it. At larger data volumes, these memory allocation and copy costs accumulate steadily.

What This Case Teaches Us

No Silver Bullets — Only Fit for Purpose

The core lesson from this experiment is: the value of any technology is always relative to the specific use case. io_uring is excellent technology, and MMAP is a time-tested solution — each excels in its own domain. The query engine's access patterns, data scale, and concurrency characteristics together determine which approach is more appropriate.

Performance Optimization Requires Measurement First

The most commendable thing this team did was actually run comparative benchmarks rather than blindly trusting the assumption that "newer technology is faster." Before you have real measurement data, any performance optimization is just a guess. As computer scientist Donald Knuth famously said: "Premature optimization is the root of all evil."

The Value of Sharing "Failures"

In the tech community, success stories are shared abundantly, while "we tried it and it didn't work" experiences are relatively rare. Sharing these negative results is equally valuable — it helps other engineers avoid repeating the same mistakes and reminds us to maintain critical thinking about technology.

Conclusion

This "failed" experiment moving from MMAP to io_uring is less a failure of io_uring and more a successful engineering validation. It shows us with real data that in the world of databases and query engines, the combination of OS page cache and MMAP remains a powerful baseline that is not easily surpassed.

For any team considering adopting a new I/O technology, this case serves as a timely reminder: before you start swapping things out, thoroughly understand your workload characteristics, then use rigorous benchmarks to validate every assumption. True performance engineering is always built on measurement, never on faith.