Moving Search Indexes to S3: Architectural Changes and Trade-offs

Title

Moving Search Indexes to S3: Architectural Changes and Trade-offs

Meta Description

Exploring how search index migration to S3 changes architecture: the elasticity and cost benefits of storage-compute separation, the costs of network latency and consistency complexity, and which scenarios truly suit a search engine on object storage.

Article Content

Putting a search index on object storage sounds like a counterintuitive engineering decision. Traditional search engines rely on low-latency random reads and writes from local disks, while S3 is known for high throughput and high durability—but carries the inherent weakness of network latency. A discussion on Hacker News raised this question: when a search index lives in S3, what architectural changes occur? Although the original post had only 3 upvotes and zero comments, the question itself points to a real and worthwhile trend in cloud-native search architecture.

Why Someone Would Want to Put the Index in S3

The core driver behind this decision is not performance but operational economics. Local SSD storage costs far more than S3, and search indexes are often massive and continuously growing. Sinking cold data or full replicas of the index into S3 can significantly reduce storage bills.

Another motivation is elasticity. Scaling a search cluster up or down usually requires data rebalancing; with local disks, this process is both slow and prone to hitting capacity limits. When the index is in S3, compute nodes become nearly stateless workers—scaling out just means launching new instances and mounting the same index data.

Durability and disaster recovery are also considerations. S3's built-in multi-replica mechanism turns index backup from an "ops task" into a "default property," making cross-AZ and even cross-region reads possible.

Search indexes and inverted indexes: The core of a search index is typically an inverted index, which maps each term appearing in documents to a list of documents containing that term (the posting list), along with term frequency, positions, and other information. Modern search engines (such as Lucene) split the index into multiple immutable segments, each being a set of small inverted index files; queries search segments in parallel and then merge results. This segment-based design is crucial for storage-compute separation: immutable segments are naturally suited for writing to object storage and are easy to evict or cache by segment. Understanding this is essential to seeing why putting the index in S3 is not about treating the entire index as one large file, but about distributing many segment files across object storage and coordinating query routing with metadata management.

The Latency Budget Is Redistributed

Once the index leaves local disk, the most direct change is that the query path gains an extra hop of network I/O. What used to be a disk read of tens of microseconds can become a millisecond-level object storage request. This means the latency budget must be redistributed: the margin left for query parsing, scoring, and ranking is compressed.

There are usually three ways to respond. The first is tiering: keep hot data in local cache or memory, place cold segments in S3, and fetch on demand. The second is prefetching and moving caching strategies forward, using S3's high throughput to batch-load before queries. The third is changing the index structure so that a single query touches fewer objects and each object is larger—reducing the number of requests rather than optimizing a single read.

These approaches are all effective, but each pays a price elsewhere—caching brings consistency complexity, prefetching brings memory pressure, and coarse-grained objects bring read amplification.

S3 request latency and billing model: S3's typical first-byte latency is on the order of tens of milliseconds, and it bills by request count (GET/PUT/LIST) and data transfer volume—completely different from the microsecond random reads and fixed hardware cost model of local SSDs. Therefore, after moving the index to S3, query latency is affected not only by network round trips but also by request volume—if a query touches a large number of small objects, even if the total data volume is not large, high-frequency requests can drive up latency and bills. This explains why engineering practice tends toward merging small segments, increasing object sizes, using batch prefetching, and introducing local cache layers to absorb repeated reads.

Decoupling Compute and Storage Becomes the Main Theme

As a storage layer, S3 naturally pushes search systems toward storage-compute separation. This brings not only elasticity but also a clearer division of responsibilities: index building and compaction can scale independently of query services, and batch jobs no longer compete with online queries for local IO.

The cost is increased system complexity. Consistency issues emerge: after index updates are written to S3, when are they visible to query nodes? An additional metadata coordination mechanism is needed. The cost model of compaction on object storage is also different—S3 bills by request, and high-frequency reads and writes of small objects quietly drive up bills.

Storage-compute separation and metadata coordination: Disaggregation of storage and compute means deploying and scaling the data persistence layer and the compute layer independently. In search systems, this means index building, compaction, and query services can run on separate node pools while sharing the same S3 data. However, object storage itself does not provide strongly consistent directory notifications or lease mechanisms, so an additional metadata service (such as etcd, ZooKeeper, or a custom coordination layer) is needed to manage segment visibility, versions, and compaction status. A common approach is to maintain "index snapshot" metadata that query nodes periodically refresh, with the write path ensuring consistency by atomically committing new snapshots.

Who It Suits and Who It Doesn't

Putting the index in S3 is better suited to systems that are read-heavy and write-light, not extremely sensitive to P99 latency, and whose index scale is large enough that storage cost becomes the primary concern. Log retrieval, archival search, and cold data analytics are typical beneficiary scenarios.

Conversely, real-time search with hard sub-millisecond latency requirements and high-frequency index updates is still better suited to local disk or in-memory solutions. Storage-compute separation is not a universal answer but an explicit trade-off: exchanging latency and complexity for cost, elasticity, and durability.

The value of this question is that it forces teams to turn "where the index lives" from a default assumption into an explicit decision—and explicit decisions are often the starting point of architectural evolution.

P99 latency and read amplification: P99 latency refers to the time within which 99% of requests complete, and is a key metric for tail latency. Object storage's latency distribution has a long tail, and occasional slow requests significantly raise P99, which greatly affects real-time search experience. Read amplification refers to the amount of data actually read to answer a query being far greater than logically needed—for example, having to read an entire large segment because of coarse object granularity. Tiered caching and coarse-grained indexing reduce request counts while potentially worsening cache consistency and read amplification. When evaluating whether an S3 solution fits, one should consider average latency, P99 tail latency, and the number of requests and read amplification factor per query—not just storage cost.

One-line Summary

Putting search indexes in S3 is a storage-compute separation trade-off: exchanging latency and complexity for cost, elasticity, and durability.

Paragraph Summary

The article explores the architectural decision to store search indexes on object storage (such as S3). The core driver is not performance but storage cost, elastic scaling, and durability. Once the index leaves local disk, the query path adds network I/O, and the latency budget must be redistributed; common responses include hot/cold tiering, prefetch caching, and coarse-grained indexing to reduce request counts. S3 pushes storage-compute separation, allowing index building, compaction, and query services to scale independently, but introduces new complexity in metadata consistency coordination and per-request billing. The conclusion: scenarios that are read-heavy and write-light, not extremely latency-sensitive, and have large index scale suit S3, while sub-millisecond real-time high-frequency search remains better suited to local disk or memory. The article emphasizes that turning index location from a default assumption into an explicit decision is the starting point of architectural evolution.