Databricks AI Search Now Natively Supports Streaming Tables and Materialized Views

Databricks has announced that its AI Search feature now natively supports Streaming Tables, Materialized Views, and Managed Tables (based on Iceberg). This update enables enterprises to build intelligent search and retrieval-augmented generation (RAG) applications directly on real-time data pipelines, without having to choose between data freshness and search performance.

From static indexes to real-time intelligence: a key evolution for AI Search

Traditional search indexes typically rely on batch synchronization, which means data often has a latency of minutes or even hours from creation to being searchable. For scenarios that require real-time decision-making—such as fraud detection, dynamic pricing, and operational alerting—this latency directly undermines the value of search. With native support for Streaming Tables and Materialized Views, Databricks AI Search now allows indexes to continuously update alongside streaming data, while leveraging the precomputation capabilities of materialized views to reduce query latency.

More importantly, this support is native, rather than pieced together through external connectors or custom ETL pipelines. Users do not need to maintain additional sync jobs; AI Search directly consumes the latest state of these tables. For teams already using Delta Live Tables or the Iceberg ecosystem, the integration cost is significantly reduced.

Streaming Tables are a table type in Databricks' lakehouse architecture used for continuously ingesting and processing streaming data, typically used in conjunction with Delta Live Tables or Apache Spark Structured Streaming. Their core characteristic is that data is written incrementally, and the table itself records data versions and checkpoints, supporting exactly-once semantics and incremental reads. Compared with regular tables, streaming tables emphasize low-latency ingestion and continuous updates, making them suitable for high-frequency changing scenarios such as event logs, clickstreams, and IoT sensor data. When AI Search natively consumes streaming tables, it can trigger index updates based on their incremental commit mechanism, without needing to poll the entire table or rely on external job scheduling—this is the key foundation for achieving near-real-time search indexes.

Materialized Views are database objects that precompute and persistently store query results. In Databricks, materialized views are typically defined by SQL and can be refreshed on a schedule or incrementally. Their value lies in completing expensive aggregation, join, or filtering operations in advance, so that queries can directly read precomputed results, significantly reducing latency and compute costs. For search scenarios, materialized views can serve as the "upstream" of an index—first cleaning, aggregating, and trimming fields from raw data before AI Search consumes it. This both controls index size and ensures query efficiency for commonly used filter conditions.

Why support for Managed Tables and Iceberg matters

Native support for Managed Tables further simplifies operations: users no longer need to manually manage the mapping between indexes and underlying storage, and AI Search automatically handles table schema changes and partition evolution. Compatibility with the Iceberg format opens the door to cross-engine collaboration—the same table can be read and written by different engines such as Spark, Trino, and Flink, and AI Search can seamlessly connect as a consumer.

This openness is particularly important for data lakehouse architectures. Enterprises often do not want search functionality bound to a single format or a single engine. With the addition of Iceberg, AI Search is more like a standard consumer within the lakehouse rather than a closed add-on component.

Apache Iceberg is an open table format that provides capabilities such as ACID transactions, schema evolution, partition evolution, time travel, and snapshot isolation for tables in a data lake. Unlike Hive tables that rely only on file directories, Iceberg manages table state through a metadata layer, enabling multiple compute engines (such as Spark, Trino, Flink, and Presto) to safely read and write the same table concurrently. This cross-engine interoperability is a core requirement of integrated lakehouse architectures. With Databricks' Managed Tables based on Iceberg, AI Search as a consumer can follow Iceberg's metadata protocol to perceive the latest snapshot of a table, thereby maintaining index consistency even when table schemas change or partitions are adjusted, reducing dependence on proprietary formats.

Practical impact on RAG and intelligent application development

One of the core pain points of retrieval-augmented generation (RAG) applications is the timeliness of the knowledge base. When underlying documents or business data change frequently, the vector index needs to be updated promptly; otherwise, the model will generate answers based on outdated information. AI Search's support for Streaming Tables directly addresses this need: after new data is written to a streaming table, the search index can reflect the changes in near real time.

Combined with Materialized Views, teams can also precompute commonly used aggregation or filtering results, controlling query costs while ensuring freshness. This combination allows RAG applications to handle high-frequency updating data sources while maintaining acceptable response latency.

Retrieval-augmented generation (RAG) is an application architecture that combines information retrieval with large language model generation: the system first retrieves relevant documents or data snippets from an external knowledge base based on the user query, then injects these snippets as context into the prompt, and the model generates the final answer. RAG's effectiveness depends heavily on the accuracy and timeliness of retrieval results—if the index lags behind the source data, the model will answer based on outdated information, producing factual errors. Therefore, the index update mechanism of a RAG knowledge base is directly related to the synchronization capability of the underlying data pipeline. AI Search's native support for streaming tables and materialized views essentially shortens the path from "data generation → searchability," enabling RAG applications to cover high-frequency updating business data sources rather than being limited to relatively static document repositories.

Practical considerations

Although native support lowers the barrier to adoption, teams still need to pay attention to several aspects: the frequency of streaming writes and the index update strategy need to match the business's actual freshness requirements; the refresh cycle of materialized views affects the real-time nature of query results; and Iceberg table metadata management may introduce additional overhead as the number of tables grows. These are not problems unique to AI Search, but they are amplified in real-time scenarios.

Overall, this update moves Databricks AI Search from a "search feature" toward a "real-time intelligent infrastructure" positioning. For teams already building data pipelines on the lakehouse, this means search capabilities can integrate more naturally into existing data flows rather than existing as an island that requires additional maintenance.

One-line Summary

Databricks AI Search now natively supports streaming tables, materialized views, and Iceberg managed tables, enabling real-time search and RAG.

Paragraph Summary

Databricks announced that AI Search now natively supports Streaming Tables, Materialized Views, and Iceberg-based Managed Tables, allowing search indexes to continuously update with streaming data without relying on batch synchronization or external ETL pipelines. This update enables enterprises to build intelligent search and retrieval-augmented generation (RAG) applications directly on real-time data pipelines, alleviating knowledge base timeliness pain points. Iceberg compatibility improves cross-engine collaboration and reduces lock-in to a single format; materialized views help balance data freshness and query costs. The article also reminds teams to consider practical factors such as streaming write frequency, materialized view refresh cycles, and Iceberg metadata overhead, and views this update as a key step in AI Search's evolution from a search feature to real-time intelligent infrastructure.