NEAREST BY Join: How Databricks Scales Vector Search for Batch Processing

Vector Search Is Moving Beyond the Serving Layer

Vector search began as a serving layer problem. The most classic use case is the chatbot: when a user asks a question, the system converts the query into a vector in real time, finds the most similar records across a massive embedding dataset, and feeds that context to a large language model to generate a response. These scenarios demand low latency and single-point queries, typically backed by dedicated vector databases or online index services.

But as retrieval-augmented generation (RAG) and embedding-based analytics have gradually penetrated enterprise data pipelines, the nature of vector search demand is shifting. More and more workloads are no longer online requests that query one record at a time — they're offline analytical tasks that compute nearest neighbors across hundreds of millions of records in a single batch. This is the core problem Databricks set out to solve by introducing NEAREST BY Join in its Runtime.

Embeddings are techniques that map text, images, or other unstructured data into a high-dimensional real-valued vector space, where semantically similar content sits closer together. Retrieval-Augmented Generation (RAG) is an architectural pattern that combines vector retrieval with large language models: before generating a response, the system first retrieves the most relevant text passages from an external knowledge base and injects them as context into the prompt. This allows the model to reference up-to-date or proprietary knowledge rather than relying solely on what was encoded in its training parameters. As RAG moves from prototype to production, enterprise knowledge bases often grow from tens of thousands of records to tens of millions. What were once single online queries evolve into offline tasks — periodically rebuilding indexes, batch-evaluating retrieval quality, running semantic deduplication across the entire corpus — exposing the limitations of vector databases originally designed for low-latency point queries when applied to batch workloads.

From Single-Point Queries to Batch Nearest-Neighbor Joins

Traditional vector databases excel at serving-layer point queries, but when you need to perform a "nearest-neighbor join" between two large vector tables, these tools often fall short. Typical batch scenarios include deduplicating an entire document corpus, finding the most similar products for each user record, or running semantic clustering and recall evaluation over large-scale datasets.

The essence of these tasks is expressing a nearest-neighbor search as a SQL-level Join operation — for each row in the left table, find the K closest records in the right table. The design philosophy behind NEAREST BY is to make vector nearest-neighbor retrieval a first-class citizen in the query engine, composable with standard operators like joins, filters, and aggregations — without having to export data to an external vector store and import the results back.

Why Batch Workloads Need a New Operator

Forcing online indexes into batch pipelines creates two major pain points. The first is scale: online vector indexes are optimized for low-latency single-record queries. When faced with full-table-scan-style joins over billions of rows, both throughput and cost become unacceptable. The second is data movement: shipping data out of the lakehouse to an external service and back again adds engineering complexity while undermining data governance and consistency. Embedding vector search natively into the runtime keeps compute close to the data and lets it reuse the engine's existing partitioning, parallelism, and resource scheduling capabilities.

The Lakehouse architecture combines the low-cost storage of a data lake with the SQL query capabilities of a data warehouse on a single platform — Databricks' Delta Lake and Apache Spark runtime are canonical implementations. In a lakehouse, data is naturally distributed in partitioned form across object storage, and the compute engine can schedule tasks to the nodes where the data resides, enabling in-place processing that keeps compute close to data. Natively integrating vector nearest-neighbor search into such a runtime means directly reusing the engine's existing mechanisms — partition pruning, vectorized execution, Adaptive Query Execution (AQE) — without maintaining a separate distributed infrastructure for vector workloads. This is especially critical for scenarios that require joint analysis across petabyte-scale structured tables and vector columns.

Nearest-neighbor search algorithms fall into two broad categories: exact and approximate. Exact K-Nearest Neighbors (KNN) requires scanning all candidate vectors, with time complexity growing linearly with data size — often unacceptable at billion-record scale. Approximate Nearest Neighbor (ANN) algorithms build index structures like HNSW (Hierarchical Navigable Small World graphs) or IVF (Inverted File Index) to reduce search complexity to sub-linear at the cost of a small precision trade-off, and are the dominant approach in industry. The challenge of Batch KNN Join is completing an independent ANN query for each row vector in the left table while maximizing overall throughput — requiring system-level coordination between partition design, index construction, and task parallelism, rather than simply looping over a single-query interface.

The Engineering Value of NEAREST BY Join

Pushing nearest-neighbor search down into the query engine layer means developers can accomplish work that previously required large amounts of glue code using familiar declarative syntax. Vector search is no longer a "side system" sitting outside the data platform — it becomes a capability that can be uniformly orchestrated alongside structured and semi-structured data within the lakehouse.

For data engineering teams, this lowers the barrier to building RAG data preprocessing pipelines, offline recall evaluation, and semantic deduplication workflows. For the platform itself, this is an important step in elevating vectors from a special data type to a general-purpose capability that can participate in large-scale distributed computation.

Closing Thoughts

The shift of vector search from real-time point queries in the serving layer to large-scale nearest-neighbor joins in the batch processing layer reflects the growing prevalence of embedding technology across enterprise data stacks. Databricks' NEAREST BY Join aims to support this need natively within the lakehouse runtime, unifying vector retrieval with traditional SQL analytics. For teams building large-scale RAG and semantic analysis pipelines, this kind of native capability is worth watching closely.

Note: This article is based on limited source material. For specific NEAREST BY syntax, performance benchmarks, and implementation details, please refer to the official Databricks technical documentation for complete information.