Lakebase Search Vector Retrieval Benchmark: Cache and Index Matching Are Key

How does search latency change when a vector index scales from one million to four million passages? The Lakebase team ran a set of benchmarks, and the conclusion is somewhat counterintuitive: what determines performance is often not the number of rows, but how well the cache size matches the index scale.

Core Findings from the Benchmark

Lakebase Search ran vector search benchmarks across a scale range of 1M to 4M passages. The core question the team focused on was: as the index grows, how does vector retrieval latency change?

The answer from the tests points to one key variable—the ratio of cache size to the vector index. When the cache roughly matches the index scale, the vast majority of page reads can avoid the underlying storage, and the p50 latency of hot vector searches stays stable between 11 and 23 milliseconds. This means query response times are at a level acceptable for interactive applications.

The value of this finding is that it shifts the focus of performance optimization from "stacking more rows" to "getting the cache ratio right."

What p50 Latency and Hot Queries Mean

p50 latency refers to the latency of the request at the median position after sorting all requests by duration—that is, half the requests are faster and half are slower than that value. Compared with the average, p50 better reflects the experience of a typical user, because the average is easily skewed upward by a small number of extremely slow requests. The "hot" queries emphasized in the article refer to the request path when index data has already been loaded into the cache and the system is in a steady operating state; they do not include cold starts, first access after cache eviction, or queries during bulk imports. Therefore, the 11 to 23 millisecond p50 should be understood as typical latency under ideal cache conditions, not a guaranteed value at any moment. When evaluating production performance, attention should also be paid to p95/p99 tail latency, because cache misses tend to concentrate in a small number of requests, and tail metrics are more sensitive to capacity planning.

Why the Cache Matters More Than the Number of Rows

Vector search is fundamentally based on nearest-neighbor retrieval, and queries require frequent access to vector data in the index. If this data resides in memory, the query path is short; once it has to fall back to the storage layer, latency rises noticeably.

Lakebase's tests show that when cache capacity is roughly aligned with the vector index scale, 95–98% of page reads avoid storage access. In other words, as long as the cache hit rate stays high, the latency of hot queries can remain low.

This explains why looking only at the number of rows can easily lead to misjudging performance. An increase in the number of rows does not necessarily worsen latency by itself; the real problem is that as the index grows larger, the cache becomes relatively smaller, the hit rate declines, and the proportion of storage reads rises.

Page Reads and the Cost of Hitting the Storage Layer

The "page reads" repeatedly mentioned in the article refer to operations in which the database reads index or data blocks from the storage layer in fixed-size data pages. Vector indexes (such as graph structures like HNSW) are organized in memory by page, and a single nearest-neighbor retrieval often requires traversing the graph to access dozens to hundreds of nodes, which may be distributed across multiple data pages. If the page is already in the in-memory buffer pool, the read takes only microseconds; once it misses and falls back to disk or remote storage, a single access may consume tens to hundreds of microseconds or even milliseconds, and this is amplified under concurrent queries. Therefore, a seemingly small change such as a cache hit rate dropping from 98% to 95% will be directly reflected in p50 and tail latency under the high-frequency random access workload of vector retrieval. This also explains why "the proportion of page reads that avoid storage access" is a better performance indicator than "the total number of rows in the index"—the former describes the actual access path, while the latter is only a static description of data scale.

Practical Implications for Production Environments

For teams running vector search, this set of benchmarks provides an actionable scaling approach: when planning dataset growth, adjust cache capacity at the same time so that it stays matched to the vector index scale.

The original conclusion is also direct—as the dataset grows, scaling cache size in sync with the vector index helps maintain low-latency search.

Interestingly, what is discussed here is the latency performance of "hot" queries, that is, the ideal state in which the cache is already ready and the index resides in memory. The original material does not elaborate on performance under cold starts or cache misses, and in actual deployment it still needs to be verified against your own workload characteristics.

Latency Budgets in RAG and Semantic Search

Retrieval-augmented generation (RAG) systems typically chain multiple stages—retrieval, reranking, and generation—within a single user request. The large model generation stage often takes hundreds of milliseconds to several seconds; by comparison, the tens of milliseconds of vector retrieval account for a small share, but they determine whether the entire pipeline can obtain context in time before the first token is output. Interactive semantic search is even more latency-sensitive, and users can perceive stutter in responses exceeding about 100 milliseconds. Keeping vector retrieval p50 at 11 to 23 milliseconds means the retrieval stage leaves ample budget for subsequent reranking and generation, as well as headroom for traffic peaks and cache jitter. This is also why the cache-to-index ratio deserves consideration during the capacity planning stage.

Summary

Lakebase Search's benchmark covers scales from one million to four million passages, and the core conclusion can be summed up in one sentence: how well the cache matches the index determines vector search latency more than the number of rows alone. For teams building retrieval-augmented generation or semantic search systems, this provides a clear reference for resource allocation.