Lakebase Search in Practice: Capacity Planning Lessons at 4 Million Passages

From Slide Hub to the Real Demands of Internal Search

How much does it actually take to build an internal search system that supports retrieval across 4 million passages? The Databricks team went through the entire journey—from prototyping to production deployment—while developing their internal application Slide Hub. Slide Hub serves field engineers, and its core requirement is to quickly locate and reuse internal documentation. When the data scale climbed to 4 million passages, the search system's latency, recall, and cost structure all changed significantly. This article distills the key decisions and lessons from capacity planning for large-scale search systems, based on the Databricks engineering team's practice.

Slide Hub's positioning is clear: let field engineers avoid digging through piles of folders and emails, and instead directly use search to find reusable technical solutions and presentation materials. The needs of such internal tools are often underestimated—users have far less tolerance for search quality than they do for consumer products. A single bad retrieval result can make them abandon the tool and return to manually sifting through files.

In the early stages, the team validated the basic retrieval pipeline with a smaller dataset. But when the document library naturally grew to the scale of 4 million passages, the index structures and query strategies that had previously been sufficient began to expose bottlenecks. This scale sits in a subtle range: it's not small data below the million level that can be brute-force scanned, nor is it heavy enough to require a distributed cluster, but it's enough to make the cost of capacity planning mistakes very concrete.

What is "chunk/passage-level retrieval"? In document retrieval, indexing entire documents often results in retrieval granularity that is too coarse—a long document may cover multiple topics, and its overall embedding dilutes the semantics. After a hit, users still have to search within the document themselves. Passage-level retrieval splits documents into fragments of a few hundred characters (chunks), each vectorized and indexed separately. The benefit is higher recall precision and results that point directly to specific content. The cost is that the number of index entries expands many times over—a document of a few thousand words may be split into a dozen or more passages. This is exactly why Slide Hub's data volume jumped from "document count" to "4 million passages," and it's the direct source of capacity planning pressure. Chunking strategy (fixed length, by heading, or by semantic boundary) significantly affects recall quality and index size.

The Art of Balancing Retrieval Quality and Latency

The trickiest part of scaling search is that there is a hard constraint between retrieval quality and response latency. The team tried various strategy combinations on Slide Hub and found that vector retrieval performs well on semantic matching, but index construction and query overhead rise non-linearly as the number of passages grows.

According to Databricks' engineering blog, once the number of passages reached 4 million, simple full vector scans could no longer meet the latency requirements for interactive search. The team had to make explicit trade-offs among recall, precision, and speed. The final solution adopted a hierarchical retrieval strategy—first using lightweight indexes to filter candidates, then finely ranking the candidate set. This architecture kept recall at an acceptable level while bringing latency into the range users can perceive as responsive.

One easily overlooked lesson: capacity planning for search systems cannot look only at storage and compute resources. Users' psychological expectation of "fast enough" is also a hard constraint that needs to be modeled.

Hierarchical retrieval (also called two-stage retrieval / recall-and-rerank architecture) is currently the mainstream paradigm for large-scale vector retrieval. The first stage uses lightweight methods to quickly narrow down candidates from the full corpus. Common approaches include hybrid recall combining inverted indexes (BM25) with approximate nearest neighbor vector search (ANN, such as HNSW or IVF-PQ). The goal is to reduce candidates to dozens or hundreds within milliseconds, sacrificing some precision for speed at this stage. The second stage uses a heavier model (such as a cross-encoder reranker) to score the candidate set precisely. Although the per-computation cost is high, the total overhead is controllable because the number of candidates is small. This combination of "coarse filtering + fine ranking" means the system does not need to apply expensive exact computation to all 4 million passages, thereby keeping P95/P99 latency within an interactively acceptable range. The core trade-off of ANN indexes lies in the triangular relationship among recall, memory, and build time.

Hidden Traps in the Cost Structure

For a search system at the scale of 4 million passages, the bulk of the cost is often not in the queries themselves, but in index maintenance and update pipelines. The Slide Hub team found that as documents continued to be added and modified, the frequency and resource consumption of index rebuilds became the main source of operational burden.

Incremental index updates are a key way to alleviate this problem, but they introduce complexity around index consistency. The team needed to find a balance between "index freshness" and "system stability": too frequent updates drive up resource consumption, while too long an interval causes search results to lag and harms user experience. This trade-off is especially delicate in internal tool scenarios—users expect to find the latest content, but they won't pay separately for search quality.

Incremental indexing and index consistency are among the most underestimated aspects of operating large-scale retrieval systems. Full index rebuilds are logically simple and guarantee consistency, but every rebuild requires recomputing vectors for all passages. At the scale of 4 million passages, this is often an hour-level batch job with enormous resource consumption that cannot be run frequently. Incremental indexing only computes embeddings for newly ingested or changed documents and appends them to the index structure, greatly reducing update costs, but it introduces several thorny problems: when deleting or updating old passages, ANN indexes (especially graph-structured HNSW) are difficult to delete from efficiently in place, so a compromise of "soft delete + periodic merge" is usually adopted; at the same time, the query side may briefly read both old and new versions of a passage, requiring version filtering or read-your-writes visibility control to ensure consistency. These complexities are precisely the technical root of the "freshness versus stability" trade-off.

A Reusable Capacity Planning Framework

From Slide Hub's practice, a planning approach for medium-scale search systems can be distilled. First, stress testing in the prototype stage must use datasets close to the real distribution; distribution bias in synthetic data can seriously mislead capacity estimates. Second, incorporate retrieval quality metrics (such as recall@K) and latency metrics (P95, P99) into the same evaluation framework to avoid suboptimal decisions caused by single-dimensional optimization. Third, reserve a separate resource quota for the index update pipeline—don't let query load and index maintenance compete for the same compute resources.

The Databricks team's experience shows that a search system at the 4-million-passage level does not require an over-engineered distributed architecture, but it does demand strong attention to detail. Choosing the right retrieval strategy, managing the index lifecycle well, and controlling tail latency determine the system's practical usability more than piling on resources. For teams building internal search tools, the Slide Hub case offers a pragmatic reference—scale is not the problem; the hidden constraints that scale brings are.

Recall@K and P95/P99 latency are two foundational metrics for evaluating retrieval systems. Recall@K refers to the proportion of relevant documents contained in the top K returned results, where K is usually 5, 10, or 20. It measures whether the results that should be found were found. Latency cannot be judged by averages alone—averages are pulled down by a large number of fast queries, masking the stalls experienced by a minority of users. P95 latency means 95% of requests complete within that time, and P99 is the 99% line. Tail latency is often the real cause of user churn: even if the average response is only 100 milliseconds, if one out of every 20 searches stalls for 2 seconds, the experience will be judged as "unstable." Putting recall@K and P95/P99 into the same evaluation framework is what prevents single-dimensional suboptimal decisions—such as "recall went up but latency collapsed"—when adjusting chunking granularity, index parameters, or candidate set size.

One-line Summary

Databricks reviews Slide Hub: lessons in capacity planning and trade-offs for internal search at 4 million passages.

Paragraph Summary

Based on the Databricks engineering team's practice developing the internal search tool Slide Hub, this article breaks down capacity planning for a search system at the scale of 4 million passages. It notes that this scale sits in a subtle range where "brute-force scanning is insufficient, but a distributed cluster is overkill," and that latency, recall, and cost structure all change significantly with data volume. Core lessons include: simple full vector scans cannot meet interactive latency requirements, so a hierarchical retrieval strategy of coarse filtering followed by fine ranking is needed; the bulk of the cost lies in index maintenance and update pipelines rather than queries themselves, and incremental indexing requires balancing freshness and stability; capacity planning should incorporate recall@K and P95/P99 latency into the same evaluation framework; and prototype-stage stress testing must use datasets close to the real distribution. The conclusion is that this scale does not require an over-engineered architecture, but the details of retrieval strategy, index lifecycle, and tail latency determine system usability.