The Case for Object Storage Supremacy: Why S3 Is Becoming the Core of Modern Data Architecture

Introduction: From "Database-Centric" to "Object Storage-Centric"

Recently, an article titled Object Storage Is All You Need sparked discussion on Hacker News. While the conversation is still in its early stages, the provocatively titled piece hits squarely on a core theme in the evolution of data infrastructure: Object storage is leaping from being a simple "cold data warehouse" to becoming the central hub of modern data architecture.

The title is clearly an homage to the seminal deep learning paper Attention Is All You Need. It attempts to convey a bold thesis — in the cloud-native era, perhaps we don't need so many complex, purpose-built storage systems. A single object storage layer that is powerful enough, cheap enough, and reliable enough could handle the vast majority of workloads.

What Is Object Storage, and Why Does It Matter?

The Essence of Object Storage

Object storage is fundamentally different from traditional block storage and file storage. It manages data as "objects," where each object contains the data itself, metadata, and a globally unique identifier. The most iconic example is Amazon S3, along with the many S3-compatible implementations (such as MinIO, Cloudflare R2, Backblaze B2, etc.).

The core advantages of object storage include:

• Near-infinite horizontal scalability: No need to plan capacity in advance — it can scale smoothly from a few GB to exabyte-level;
• Extremely low storage costs: Per-unit storage costs are typically an order of magnitude lower than block storage;
• High durability and availability: Major cloud providers commonly guarantee 99.999999999% (eleven 9s) data durability;
• Standardized HTTP API: Accessible through simple RESTful interfaces, making it a natural fit for distributed and cloud-native environments.

From the Periphery to the Core

Historically, object storage was seen as an "archival tier" — a place to park logs, backups, images, videos, and other unstructured data. Meanwhile, databases, caches, message queues, and other "hot" components ran on expensive local disks or block storage.

But this landscape is shifting. An increasing number of modern systems are adopting object storage as their primary storage layer. Through a "separation of storage and compute" architecture, compute nodes become stateless, elastically scalable ephemeral resources, while all persistent state is pushed down into object storage.

Separation of Storage and Compute: The Underlying Logic Behind Object Storage's Rise

Restructuring the Cloud-Native Data Stack

Behind the "object storage is everything" thesis lies a profound restructuring of the entire data stack. Take the data analytics space as an example: Snowflake's success is largely attributable to its separation of storage and compute — data is stored in columnar formats on S3, and compute clusters are spun up on demand and shut down when idle. This pattern has since been further popularized by Databricks' Lakehouse architecture, Apache Iceberg, Delta Lake, and other "open table formats."

Today, this trend has spread into even more domains:

• OLTP databases: Next-generation databases like Neon and TiDB are exploring building their storage layers on top of object storage;
• Message queues: Confluent's Kafka has introduced tiered storage, offloading historical messages to S3; WarpStream has gone even further, building a fully S3-based, local-disk-free Kafka-compatible system;
• Search and observability: Systems like Quickwit and GreptimeDB also use object storage as their core storage medium.

Driven by Both Cost and Elasticity

The fundamental driver behind this trend is economics. Object storage isn't just cheap — more importantly, it completely decouples the cost models of "storage" and "compute." Organizations no longer need to buy excess compute capacity just to scale storage, or vice versa. In an era of explosive data growth and cyclical compute demand, the cost savings from this elasticity are enormous.

Challenges and Boundaries: The Price Behind Object Storage "Supremacy"

Despite the appealing outlook, the "object storage is everything" thesis is far from uncontroversial. Using object storage as primary storage requires tackling several fundamental technical challenges:

Latency

Object storage access latency typically falls in the range of tens to hundreds of milliseconds — far higher than the microsecond-level latency of local NVMe SSDs. For OLTP scenarios that demand low-latency random reads and writes, this is a serious challenge. Engineers must introduce multi-tier caching, prefetching, batching, and other complex mechanisms to bridge this gap.

Consistency Model

Although S3 has offered strong consistency since 2020, object storage lacks native transaction support. Systems built on top of it must implement their own concurrency control, atomic commits, and similar semantics, significantly increasing system design complexity.

The Cost Trap of High-Frequency Small Objects

Object storage API requests (PUT/GET/LIST) are individually priced. When workloads involve high-frequency access to large numbers of small objects, request fees can far exceed storage costs, making it paradoxically expensive. This requires upper-layer systems to carefully design data layouts, aggregating small objects into larger ones to reduce request overhead.

Conclusion: A Pragmatic Kind of "Supremacy"

Let's return to that title — Object Storage Is All You Need. Much like the paper it pays homage to, the absolutism of the title is more of a rhetorical device than a rigorous technical assertion. Object storage can't truly replace everything — local caching, in-memory computing, and specialized index structures will remain indispensable for the foreseeable future.

But the thesis captures a real and important trend: Object storage is becoming the unshakeable foundation of modern data architecture. More and more systems are choosing to entrust their persistent state to it, then building lightweight, elastic, stateless compute layers on top. This "object storage-centric" design philosophy is quietly reshaping how we build data-intensive applications.

For architects and engineers, understanding and leveraging the capabilities and boundaries of object storage may well be the key to mastering the next generation of cloud-native systems.